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capability  of  altering  their  user  interface.  The  military  will  gladly  pay  for  additional 
peripheral  devices  but  only  if  they  can  demonstrate  improved  training 
effectiveness.  This  research  project  seeks  to  establish  an  input  device 
configuration  solution  for  virtual  desktop  trainers.  Specifically,  we  compared  the 
standard  laptop  keyboard  and  mouse  to  a  configuration  incorporating  a  game 
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I.  INTRODUCTION 


A.  PROBLEM  STATEMENT 

Personal  computer  and  console  gaming  technology  has  advanced  to  the 
point  where  virtual  desktop  trainers  have  become  ubiquitous  to  the  U.S.  military. 
Numerous  training  transfer  studies  have  been  conducted  demonstrating  that 
personal  computer  and  console  gaming  technologies  are  at  least  as  effective  as 
traditional  training  methods  (Brown,  2010).  Virtual  desktop  trainers  have  the 
capability  of  altering  the  interface  by  adding  game  controllers,  larger  screens, 
head  mounted  displays,  head-tracking  devices,  and  improved  communication 
interfaces.  Recent  virtual  desktop  training  transfer  studies  have  recommended 
investigating  the  effectiveness  of  these  peripheral  devices  to  determine  whether 
they  add  enough  training  benefit  to  justify  their  cost  and  added  logistical  burden. 
The  military  will  gladly  pay  for  additional  peripheral  devices  but  only  if  they  can 
demonstrate  an  improvement  on  training  effectiveness. 

This  thesis  seeks  to  recommend  a  configuration  solution  and  provide 
Commanders  and  Battle  Command  Training  Center  (BCTC)  directors  a  reference 
when  making  their  virtual  desktop  trainer  peripheral  interface  decisions. 
Specifically,  we  compared  the  standard  laptop  keyboard  and  mouse  as  user- 
input  device  to  a  configuration  incorporating  a  game  controller.  Additionally,  we 
compared  the  standard  laptop  configuration  to  a  configuration  incorporating  a 
head-tracking  device. 

We  defined  the  problem  as:  “When  using  a  virtual  desktop  simulation  for 
training,  do  commercial  head-tracking  and  video  game  controller  devices  improve 
training  effectiveness  by  providing  a  more  intuitive  user  interface?” 

A  more  intuitive  user  interface  could  minimize  the  initial  training  time 
required  to  teach  users  to  perform  basic  individual,  weapon,  and  vehicle 
functions  in  a  virtual  desktop  simulator.  This  time  savings  would  result  in  more 
time  available  to  train  the  skills  that  will  keep  a  soldier  alive  in  combat. 
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This  thesis  project  aims  to  gain  insight  into  where  the  trainer  should  invest 
his  money.  The  military  willingly  spends  money  to  offer  the  best  possible  training 
to  Soldiers  and  Marines,  but  the  extra  expense  must  have  proven  worth.  One 
such  expense  is  peripheral  devices  used  in  conjunction  with  Virtual  Battlespace  2 
(VBS2™).  Peripheral  devices  considered  in  this  study  have  the  potential  to 
leverage  recent  advances  in  gaming  technology  and  trainee  gaming  experience. 
These  peripheral  devices  could  improve  individual  gaming  performance  and 
minimize  the  time  required  to  gain  sufficient  gaming  proficiency.  This  would 
result  in  more  time  dedicated  to  training  military  skills  using  the  game  as  a  tool 
instead  of  learning  how  to  play  the  game.  The  research  team  proposes  to  gain 
insight  into  user-input  interfaces  by  investigating  both  performance  effects  and 
subjective  preference. 

B.  RESEARCH  QUESTIONS 

1.  Do  user-input  devices  impact  military  training  performance  when 

used  with  a  desktop  virtual  training  simulation? 

2.  Do  soldiers  prefer  one  user-input  device  over  another  when  using  a 
desktop  virtual  training  simulation? 

C.  HYPOTHESES 

1.  The  addition  of  a  commercial  head-tracking  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training  simulation  and 
be  preferred  over  no  head-tracking  device. 

2.  The  addition  of  a  commercial  game  controller  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training  simulation  and 
be  preferred  over  the  standard  keyboard  and  mouse. 

D.  EXPLORATORY  HYPOTHESES 

Do  factors,  such  as  age  and  map  representation  influence  the  effect  of 
user  input  device  on  performance  and  preference? 
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E.  MOTIVATION 

The  U.S.  Army  and  Marines  are  currently  fielding  VBS2™  to  Forces 
Command  (FORSCOM)  units,  Training  and  Doctrine  (TRADOC)  school  houses, 
and  Battle  Command  Training  Centers  (BCTC).  The  current  VBS2^''^ 
configuration  utilizes  a  standard  keyboard  with  no  game  controller  or  head¬ 
tracking  options.  The  U.S.  Marine  Corps  currently  deploys  VBS2™  as  part  of  the 
Deployable  Virtual  Training  Environment  (DVTE).  DVTE  is  a  laptop-based 
platform  for  a  wide  variety  of  training  simulations  in  the  United  States  Marine 
Corps  (USMC).  The  U.S.  Army  typically  installs  VBS2™  on  standard  desktop 
personnel  computers  (PC).  In  summary,  both  services  are  fielding  the  system 
using  a  keyboard  as  the  user-input  interface  with  no  game  controller  or  head¬ 
tracking  options. 

In  order  for  the  simulation  to  be  useful.  Soldiers  and  Marines  must  use  the 
interface  to  proficiently  drive  vehicles,  shoot  weapons,  and  maneuver  their 
bodies  as  they  would  in  the  real  world.  This  study  sought  to  determine  whether 
the  game’s  keyboard  and  mouse  interface  was  comparable  to  using  a 
PlayStation  3  game  controller. 

VBS2™  has  the  capability  to  change  the  user-input  interface  from 
keyboard  to  the  Sony  Playstation  3  (PS3™)  game  controller  and  to  integrate  with 
NaturalPoint’s  TrackIR  5  head-tracking  technology.  The  PS3™  game  controller 
is  ergonomically  engineered  to  be  a  more  intuitive  device  to  control  a  playable 
character  or  vehicle  in  a  video  game.  NaturalPoint’s  TrackIR  5  head-tracking 
technology  allows  the  user  to  change  the  player's  view  perspective  in  the  virtual 
environment  by  simply  moving  their  head. 

Unit  Commanders  and  BCTC  Directors  need  to  decide  how  to  configure 
VBS2™  user-input  interfaces,  in  order  to  maximize  the  VBS2™  training 
experience.  This  research  project  seeks  to  recommend  an  optimum 
configuration  that  minimizes  the  training  time  required  to  teach  users  to  perform 
basic  individual,  weapon,  and  vehicle  function  in  VBS2™.  By  reducing  this  initial 
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training  time,  we  will  maximize  the  amount  of  time  a  Soldier  or  Marine  can  spend 
focusing  on  military  training  instead  of  learning  a  game. 

F.  SOLDIER,  SAILORS,  MARINES,  AIRMEN,  AND  COAST  GUARDSMEN 

This  thesis  research  project  was  originally  conceived  by  a  United  States 
Marine  Corps  Officer  but  planned  and  executed  by  a  United  States  Army  Officer. 
The  study  participants  were  officers  and  enlisted  service  members  attending  the 
Defense  Language  Institute  or  the  Naval  Postgraduate  School  representing 
every  U.S.  Armed  Service.  There  were  even  international  service  members 
represented  as  well. 

This  was  truly  a  joint  operation  with  all  military  services  well  represented. 
From  this  point  on,  we  will  simply  use  the  term  “Soldier”  when  referring  to  all 
service  members,  all  of  which  have  given  some,  and  some  will  be  asked  to  give 
all. 

G.  SCOPE  OF  THE  THESIS 

The  primary  purpose  of  this  thesis  is  to  determine  the  correct  combination 
of  peripherals  to  maximize  the  VBS2™  training  experience.  This  thesis  will  focus 
on  user-input  devices.  Specifically,  it  will  focus  on  how  different  combinations  of 
user  input  devices  affect  individual  mounted  and  dismounted  movement  within 
the  VBS2™  virtual  training  environment,  ability  to  accurately  identify  and  engage 
targets  with  personal  and  crew  served  weapons.  The  scope  of  this  thesis  will  not 
include  the  value  of  specific  peripherals  in  training  collective  task  (team)  training. 

H.  BENEFITS  OF  STUDY 

This  thesis  seeks  to  establish  a  recommended  user  input  device 
configuration  for  virtual  desktop  trainers  such  as  VBS2™.  It  will  also  serve  as  a 
reference  to  BCTC  directors  when  making  their  VBS2™  peripheral  interface 
decisions.  This  guidance  will  be  supported  with  research  data  that  will 
demonstrate  cause  and  effect  outcome  of  said  use.  A  main  benefit  of  this  study 
is  the  potential  time  saving  in  training  and  subsequent  increase  in  important 
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military  training.  It  also  has  the  potential  to  address  who  and  under  what 
conditions  will  most  benefit  from  the  recommended  configuration. 

I.  THESIS  ORGANIZATION 

Chapter  I:  Introduction 
Chapter  II:  Background 


Chapter  III: 

Methodology 

Chapter  IV: 

Pilot  Study  Experiment 

Chapter  V: 

Defense  Language  Institute  Experiment 

Chapter  VI: 

Recommendations  and  Conclusions 
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DLI  Post  Exercise  Survey 
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DLI  Post  Exercise  Survey  Addition 
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DLI  Study  maps 
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II.  BACKGROUND 


A.  INTRODUCTION 

Today,  the  U.S.  military  operates  in  an  environment  that  is  characterized 
by  uncertainty,  complexity,  and  rapid  change.  To  prevail  in  this  environment,  the 
United  States  Armed  Forces  must  be  capable,  against  a  plethora  of  current 
threats,  adaptable  to  rapidly  emerging  new  threats,  and  ready  to  respond  across 
the  full  range  of  military  operations.  The  non-state,  insurgent  and  terrorist 
adversaries  the  nation  currently  faces  around  the  world  have  chosen  asymmetric 
approaches  to  warfare  that  avoid  the  conventional  strengths  of  the  United  States 
Armed  Forces.  To  counter  these  threats,  the  military  must  remain  creative  and 
flexible  if  we  are  to  confound  our  enemies’  designs. 

Training  with  virtual  simulations  offers  great  potential  to  keep  the  military 
units  at  its  maximum  effectiveness  by  instilling  adaptability  and  flexibility  before 
the  force  is  engaged  in  combat.  By  exposing  leaders  at  every  level  to  the 
complex  operational  environment,  training  with  virtual  simulations  offers  an 
efficient  and  effective  way  to  increase  the  readiness  of  the  United  States  Armed 
Forces  from  the  headquarters  to  the  small  unit  and  individual  level. 

Across  all  warfighting  communities,  simulated  training  advances  have 
been  significant,  yet  the  use  of  advanced  simulation  technology  has  not  been 
achieved  for  the  training  of  infantry  small  units  in  close  combat.  State-of-the-art 
simulation  training  that  is  demanded  and  accepted  as  routine  for  aviation,  armor, 
or  maritime  forces,  is  negligible  or  almost  non-existent  on  a  large  scale  for  U.S. 
ground  forces. 

Since  1945,  American  infantry  units  have  suffered  over  80  percent  of  our 
nation’s  military  casualties.  Research  concludes  these  casualties  often  occur  in 
the  initial  fire  fights,  yet  very  few  resources  have  been  applied  to  the 
development  of  realistic  immersive  simulation  of  ground  operations  to  prepare 
ground  troops  for  their  first  engagements  with  the  enemy.  Though  the 
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rudimentary  simulation  designed  for  close  combat  currently  affords  units  some 
level  of  challenge,  it  does  not  yet  approach  the  level  of  sophistication  that  is 
commonplace  and  deemed  essential  in  the  other  warfighting  disciplines. 

B.  VIDEO  GAMES  AND  THE  U.S.  ARMY 

In  1980,  a  group  of  retired  general  grade  officers  approached  Atari^'^  with 
the  idea  that  the  technology  form  the  Atari  game  Battlezone  could  be  used  to 
train  soldiers  for  the  then  new  Infantry  Fighting  Vehicle.  Over  the  next  year.  Atari 
transformed  Battlezone  into  a  military  training  device.  At  the  1981  Worldwide 
Training  and  Doctrine  (TRADOC)  Conference,  the  new  training  simulator  was 
unveiled.  The  U.S.  Army  was  beginning  to  recognize  that  future  Soldiers  would 
grow  up  in  an  environment  of  electronic  gadgetry,  and  teaching  methods  for  this 
new  breed  had  to  be  developed  accordingly. 

During  the  1981  Worldwide  TRADOC  Conference,  General  Donn  Starry 
asked  the  question  “In  an  era  that  has  seen  such  fantastic  technological 
achievements,  how  is  it  that  our  Soldiers  are  still  sitting  in  classrooms,  still 
listening  to  lectures,  still  depending  on  books  and  other  paper  reading  materials, 
when  possibly  new  and  better  means  for  training  have  been  available  for  many 
years?”  (Halter,  2006).  General  Starry  asked  a  question  that  is  still  pertinent 
more  than  a  quarter  century  later. 

The  first  three  decades  of  modern  computer  history  produced  machines 
so  expensive  and  cumbersome  that  they  remained  the  domain  of  large  funding 
efforts,  such  as  Defense  Department  projects.  Through  the  Cold  War,  simulation 
enabled  blue  icons  and  red  icons  to  march  across  screens  toward  each  other 
while  numerous  mathematical  calculations  cranked  out  answers  to  who  won  and 
lost  each  engagement.  Post  Cold  War  strategy  and  tactics  changes  suggested 
the  need  for  new  types  of  simulation.  The  military  looked  around  and  saw  a 
civilian  gaming  market  flooded  with  new  and  interesting  technology. 

After  the  initial  2003  invasion  into  Iraq,  the  U.S.  Army  found  their  biggest 

weakness  was  their  ability  to  safely  conduct  logistical  convoys.  The  U.S.  Army 
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did  not  have  a  convoy  trainer  so  they  asked  the  Defense  Advanced  Research 
Project  Agency  (DARPA)  for  assistance.  In  2004,  DARPA  went  to  the  civilian 
gaming  market  for  an  answer,  where  they  found  Operation  Flashpoint.  DARPA 
initiated  a  variety  of  changes  to  make  the  game  military  and  realistic  and 
produced  DARWARS  Ambush!  (Peck,  2004).  Soon  the  Army  used  the 
simulation  both  in  theater  and  stateside  to  enhance  training.  Additionally, 
personnel  began  to  experiment  using  the  game  for  rehearsal  with  Soldiers  at  Fort 
Polk  providing  virtual  tours  of  the  places  where  troops  would  fight  weeks  later 
(Laurent,  2007). 

In  August  2008,  Smith  responded  to  an  interview  question  about  modern 
Soldiers’  familiarity  with  computer  games  “Our  research  and  hands-on 
experience  shows  that  about  50  percent  of  young  enlisted  Soldiers  call 
themselves  “gamers”  or  are  familiar  with  the  mechanics  of  game  play.  At  the 
officer  level,  it  is  around  33  percent.  We  have  learned  that  we  cannot  assume 
that  all  Soldiers  have  this  familiarity”  (Atkinson-Bonasio,  2008).  While  the  wired 
generation  may  know  cell  phones  and  iPods,  they  do  not  necessarily  know  how 
to  use  the  games  that  support  military  training.  Moreover,  military  training  cannot 
leave  the  50  percent  or  any  other  percentage  of  non-gamers  behind. 

On  5  January  2009,  the  U.S.  Army  announced  Bohemia  Interactive  was 
the  prime  contractor  for  a  new  training  program  known  as  “Game  After  Ambush”. 
The  Program  Executive  Office  for  Simulation,  Training,  and  Instrumentation 
(PEOSTRI)  awarded  the  contract  to  provide  an  enterprise  level  license  for  a 
desktop  training  simulator  to  the  U.S.  Army.  It  also  stated  that  Bohemia 
Interactive’s  “Virtual  Battle  Space  2”  (VBS2  ™)  is  the  base  platform  to  satisfy  the 
U.S.  Army’s  desktop  virtual  training  environment  needs.  VBS2™  has  already 
been  adopted  by  the  U.S.  Marine  Corps,  and  the  Armed  Forces  of  the  United 
Kingdom,  Australia,  and  New  Zealand.  Today,  there  are  over  10,000  U.S.  Army 
personal  computers  with  VBS2™  installed  for  training  purposes. 
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C.  VIRTUAL  DESKTOP  TRAINERS 

As  the  civilian  computer  gaming  market  has  matured  its  product  line, 
military,  medical,  educational,  and  other  communities  have  considered  the 
training  value  of  this  software.  The  military  uses  civilian  games  for  training  and 
analysis.  To  distinguish  computer  gaming  with  an  actual  military  purpose  from 
typical  civilian  games,  Ben  Sawyer,  a  high-tech  freelance  writer  and  technology 
consultant  coined  the  term  “virtual  desktop  trainers,”  and  the  term  has  caught  on 
(Macedonia,  2005).  Two  Canadian  researchers  summarized  potential  uses  for 
Virtual  desktop  trainers  in  a  paper  prepared  for  the  2008  Interservice/Industry 
Training,  Simulation,  and  Education  Conference  (l/ITSEC).  These  uses  include 
showing  viewpoints  of  opposing  forces,  teaching  action  drills,  improving  post¬ 
exercise  analysis  with  different  perspectives,  and  representing  future 
technologies  not  yet  available  for  conventional  training  (Roman  &  Brown,  2008). 

Virtual  desktop  trainers  face  a  potential  drawback  because  of  their 
interfaces.  Game  manufacturers  expend  much  effort  producing  the  most 
intuitive,  efficient,  and  user-friendly  interface  for  their  software,  but  for  any  game, 
users  must  learn  the  interface.  When  training  with  virtual  desktop  trainers, 
military  personnel  must  move,  shoot,  and  communicate  with  the  same  ease  as 
they  do  in  live  training  events.  The  game  interface  must  serve  as  an  extension  of 
the  service  member’s  warfighting  skills.  Interface  training  clouds  the  potential 
gain  of  virtual  desktop  trainers’  ability  to  train  because  of  the  extra  time  required 
and  the  artificiality  and  the  non-veridicality  of  the  interface.  If  the  individual  did 
not  have  to  learn  the  game,  he  or  she  could  do  something  more  productive.  The 
military  does  not  need  games  for  gaming  sake;  gaming  only  serves  as  a  means 
to  an  end.  With  this  in  mind,  military  virtual  desktop  trainers  endeavors  should 
strive  to  minimize  time  lost  to  learning  the  interface. 

D.  THE  INTERFACE  ISSUE 

The  game  interface  issue  has  bubbled  under  the  surface  of  the  civilian 
gaming  community  for  years.  Civilian  gaming  does  not  have  performance  goals 
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per  se;  rather,  goals  orient  around  entertainment  and  money.  In  many  ways,  the 
civilian  game  interface  issue  gets  to  the  crux  of  the  choice  between  console  and 
personal  computer  based  games.  Most  American  households  sport  at  least  one 
personal  computer,  so  the  game  console  is  an  extra  machine  in  the  household. 
Gamers  need  an  incentive  to  buy  that  extra  equipment,  so  the  high  dollar  console 
gaming  industry  has  a  direct  interest  in  keeping  players  on  game  controllers. 
This  interest  goes  so  far  as  to  design  the  game  to  optimize  the  effects  of  the 
controller.  Little  research  has  been  done  on  which  interface  is  better,  but  the 
Internet  is  full  of  user  opinion  on  the  subject.  For  example,  blogger  D.  Coldewey 
(2009)  wrote  a  recent  discourse  on  the  virtues  of  a  personal  computer  interface 
in  first  person  shooters,  summing  up  by  stating  that  the  interface  performance  is 
platform  dependent,  and  thus  a  platform  issue  despite  his  keyboard  interface 
preference. 

These  civilian  game  interface  design  ideas  typically  orient  on 
entertainment  value.  The  military,  on  the  other  hand,  value  realism  over 
entertainment.  Transparent  extension  of  warfighting  skills  is  the  key  determinant 
of  game  value.  In  this  respect,  performance  in  the  long  run  is  not  as  much  a 
consideration  as  the  ease  with  which  the  average  Soldier  can  learn  the  interface 
and  get  down  to  effective  training.  The  interface  must  be  as  intuitive  to  a  novice 
as  it  is  effective  in  facilitating  training  effect  in  the  game. 

E.  EVOLUTION  OF  GAME  CONTROLLERS 

A  controller  is  a  device  used  with  games  or  entertainment  system  to 
control  a  playable  character,  vehicle,  object,  or  otherwise  influence  the  events  in 
a  computer  or  video  game.  It  is  the  way  a  player  communicates  with  the  game. 
Some  examples  of  controllers  are  keyboards,  mice,  game  controllers,  joysticks, 
steering  wheels  for  driving  games,  and  light  guns  for  shooting  games.  The  type 
of  controller  used  depends  on  the  genre  of  the  game  being  played. 
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A  game  controller  is  a  special  type  of  controller  held  in  the  hand,  where 
the  digits  (especially  thumbs)  are  used  to  provide  input.  Game  controllers 
generally  feature  a  set  of  action  buttons  handled  with  the  right  thumb  and  a 
direction  controller  handled  with  the  left.  Modern  game  controllers  can  all  trace 
their  layout  and  functionality  to  the  Super  Nintendo  Entertainment  System 
(SNES)  game  controller,  which  in  the  early  1990s  popularized  the  layout  used  by 
most  modern  game  controllers. 


Figure  1 .  Super  Nintendo  Entertainment  System  (SNES)  game  pad  (From 

Mitchell,  2010) 

Some  common  additions  to  the  standard  game  controllers  include 
shoulder  buttons  placed  along  the  edges  of  the  pad,  centrally  placed  start,  select, 
and  mode  buttons,  and  an  internal  motor  to  provide  force  feedback. 

Over  the  past  forty  years,  video  game  controllers  have  gone  through  many 
changes  that  have  made  controlling  a  character  easier  and  playing  the  game 
more  enjoyable.  Some  of  these  changes  include  the  design  of  controllers,  the 
controller  interface,  and  ergonomic  changes  made  to  controllers. 

In  the  early  1970s,  when  the  first  Atari™  units  came  out,  the  controller  was 
actually  built  into  the  console.  This  design  meant  that  players  had  to  sit  by  the 
console  to  play.  Aside  from  being  annoying,  it  also  presented  another  problem. 
If  a  controller  was  to  break,  or  malfunction,  then  the  entire  system  had  to  be 
brought  in  for  repair.  The  Atari  2600™  was  one  of  the  first  systems  that  allowed 
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you  to  disconnect  the  controller  from  the  console.  This  feature  allowed  players 
more  freedom  while  playing  the  game  and  made  it  easier  to  fix  controllers.  As 
the  video  game  controller  continued  to  evolve,  new  innovations  were  introduced, 
such  as  longer  connector  cables,  cable  extensions,  and  wireless  controllers 
(Watcher,  2010). 
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Figure  2.  Evolution  of  Console  Game  Controllers  Over  Time  (From  Lopez, 

2010) 

Video  game  controllers  went  through  a  vast  change  in  the  way  players 
communicated  with  the  console.  In  the  beginning,  controllers  used  knobs  in 
order  to  move  the  character  up  or  down,  and  eventually  left  or  right.  Knobs  were 
later  replaced  by  the  invention  of  the  joystick.  The  joystick  allowed  players  to 
move  a  character  in  multiple  directions  simply  by  moving  the  joystick.  A  single 
action  button  also  accompanied  joysticks.  This  single  action  button  later  turned 
into  multiple  buttons.  The  downsides  to  the  joystick  were  minor  strain  on  the 
wrist  and  they  were  easily  broken  due  to  excessive  force  from  the  player.  During 
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the  80s  and  90s  the  invention  of  the  digital  pad  (D-Pad)  changed  controller 
interface  for  the  better.  It  allowed  players  to  have  the  same  motion  as  the 
joystick,  without  the  hassles  the  joystick  presented.  The  D-Pad  also  allowed  for 
the  easy  use  of  multiple  buttons,  and  the  functions  of  slow  motion  and  rapid-fire 
simulation.  In  the  mid  to  late  90s,  two  controllers  came  out  that  revolutionized 
gaming.  These  two  game  controllers  were  the  Playstation  and  Nintendo-64 
game  controllers. 

The  original  Sony  PlayStation  was  released  in  1995  and  its  game 
controller  had  a  cross-shaped  button  layout  and  also  had  four  shoulder  buttons. 
This  total  of  8  action  buttons  allowed  players  control  in  unprecedented  ways. 
Also,  along  with  a  D-Pad,  two  analog  sticks  move  a  character  like  never  before. 
These  sticks  allowed  a  player  to  control  the  speed  of  a  character  by  controlling 
the  amount  of  pressure  applied  to  the  stick  (Watcher,  2010). 


Figure  3.  Original  Playstation  Game  Controller  (From  Mitchell,  2010) 

The  Nintendo-64  was  released  in  1996,  and  its  game  controller  featured 
many  of  the  same  innovations  with  a  somewhat  different  layout.  Instead  of  two 
analog  sticks,  the  Nintendo-64  controller  only  had  one  analog  stick.  Also,  the 
controller  featured  six  face  buttons  with  two  shoulder  buttons.  These  two 
controllers  lead  to  the  modern  controllers  that  we  have  today  (Watcher,  2010). 
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Figure  4.  Nintendo-64  Game  Controller  (From  Mitchell,  2010) 


Most  of  today’s  modern  controllers  provide  the  same  features  as  the 
original  PlayStation  game  controller.  Face  buttons  and  shoulder  buttons  are 
used  for  character’s  actions  and  two  analog  sticks  and  a  D-Pad  is  used  for  a 
character’s  movements.  Flowever,  today’s  modern  day  controllers  feature  some 
ergonomic  changes  that  make  playing  video  games  much  easier  and  more 
enjoyable.  While  the  Sony  PlayStation  3  game  controller  is  almost  identical  to  its 
predecessor,  the  Xbox  360  game  controller  features  a  design  that  fits  the  players 
hands.  Both  of  these  controllers  use  recent  technology  such  as  Wi-Fi  and 
Bluetooth  that  allow  for  wireless  gaming  and  a  long  lasting  battery  life  (Watcher, 
2010). 


Figure  5.  Sony  DualShock  PS3™  and  Microsoft  Xbox  360™  Game 

Controllers  (From  IGN,  2010) 
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Video  game  controllers  have  expanded  into  using  infrared  (IR)  technology 
to  track  player's  head  movements.  These  head  movement  are  then  translated 
into  the  game  by  changing  the  players'  visual  perspectives.  Over  the  years, 
video  game  controllers  have  evolved  from  primitive  knobs  to  advanced 
controllers  using  a  variety  of  buttons,  sticks,  and  body  tracking  for  better  gaming 
experience. 

F.  USE  OF  GAME  CONTROLLERS  OUTSIDE  OF  VIDEO  GAMES 

1.  Introduction 

The  evolution  and  improvements  of  game  controllers  have  resulted  in 
other  industries  leveraging  the  advantages  of  game  controllers  to  manipulate 
more  than  just  games.  The  examples  are  numerous  but  I  will  focus  on  how  the 
military  is  leveraging  the  experience  of  soldier's  playing  video  games  to  better 
equipment  them  in  performing  their  war-time  duties.  The  following  are  just  a  few 
examples  of  where  game  controllers  are  finding  homes  outside  of  video  games. 

2.  Office  of  Naval  Research 

In  May  2008,  the  Office  of  Naval  Research  (ONR)  hosted  “Fleet  Week”  in 
New  York.  This  “Fleet  Week”  allowed  some  of  the  writers  from  Popular 
Mechanics  magazine  to  check  out  some  of  the  new  weapons  the  Office  of  Naval 
Research  was  developing.  One  of  the  new  weapons  system  on  display  was  the 
new  Lightweight  Stabilized  M240  Weapon  System.  This  system  is  a  swiveling 
rooftop  gun  mount  for  Highly  Mobile  Multi-Wheeled  Vehicle  (HMMWV)  and  is 
completely  controlled  from  inside  the  vehicle.  One  Popular  Mechanics  writer 
commented  on  how  similar  the  sighting  system  was  to  a  video  game.  The  ONR 
representative  agreed  and  then  proceeded  to  demonstrate  an  alternate  weapons 
control  system  that  utilized  a  XBOX  360  game  controller  to  manipulate  and  sight 
the  Light  Weight  Stabilized  M240  Weapon  System.  This  system  has  been 
fielded  and  deployed  to  service  members  serving  in  Iraq  and  Afghanistan 
(Derene,  2008). 
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3.  Improvised  Explosive  Devices  (lED)  Disposal  Robots  and 
Unmanned  Ground  Vehicle  Robots 

The  fight  to  counter  the  deadly  lED  in  Iraq  and  Afghanistan  has  had  few 
highs  and  too  many  lows.  One  of  the  bright  spots  is  the  development  and 
fielding  of  I  ED  disposal  robots  such  as  the  Talon,  manufactured  by  Foster-Miller, 
and  iRobot's  Packbot.  They  have  both  proved  their  worth,  and  saved  lives  by 
lessening  the  exposure  of  Explosive  Ordinance  Disposal  personnel  to  bombs. 
Byron  Brezina,  robotics  director  of  Naval  explosive  ordnance  technology  division, 
thinks  that  this  technology  is  great  “but  we  can  do  better”  (Magnuson,  2008). 

One  of  the  recent  improvements  is  the  addition  of  game  controllers  that 
closely  resemble  the  Sony  PlayStation  game  controller.  One  of  the  driving 
factors  in  choosing  the  game  controller  was  that  “it  is  an  intuitive  system  for  those 
who  grew  up  playing  video  game”  according  to  Kevin  Harrington,  an  account 
manager  at  iRobot  (Magnuson,  2008).  According  to  Harrington,  it  takes  less  than 
10  minutes  to  learn  to  drive  the  robot  regardless  of  any  video  game  experience. 
“All  you  need  is  two  thumbs  and  a  couple  fingers  for  this”  (Magnuson,  2008). 
Both  the  Talons  and  the  Packbots  have  fielded  the  new  PlayStation-like 
controller  as  part  of  the  user-input  device.  The  commander  of  Task  Force  Troy, 
the  U.S.  military's  in-country  counter-IED  team,  said  the  game  controllers  were 
well  received  by  the  I  ED  disposal  robot  operators.  “Soldiers  play  a  lot  of  games 
when  they’re  not  working.  It  was  very  intuitive  for  them.” 

Defense  Advanced  Research  Projects  Agency  (DARPA)  also  has  seen 
the  potential  of  game  controllers  outside  of  video  games.  As  part  of  the  U.S. 
Army's  and  DARPA's  Unmanned  Ground  Vehicle  program,  Lockheed  Martin 
developed  the  Multifunction  Utility/Logistics  and  Equipment  (MULE)  system  to 
transport  equipment  over  complex  terrain  and  obstacles  that  a  dismounted  squad 
will  encounter  during  combat  operations.  When  reporters  from  Popular 
Mechanics  visited  the  developers  of  the  MULE  at  a  test  track  in  Grand  Prairie, 
Texas,  they  saw  the  MULE  operator  using  a  XBOX  360  game  controller  to  drive 
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the  vehicle.  The  system  is  still  in  the  development  phase  but  engineers  predict 
some  variation  of  game  controllers  will  be  used  as  the  primary  user-input  device 
(Sofge,  2009). 

4.  Raytheon  System  Development 

Mark  Bigham,  director  of  business  development  for  Raytheon  Tactical 
Intelligence  Systems,  had  this  to  say  about  game  controller  expanding  out  of  the 
video  game  community,  “There  are  a  lot  of  important  lessons  to  learn  from  the 
gaming  community”  (Derene  2008).  Bigham  then  explains  how  in  the  past,  the 
military  far  outspent  the  gaming  industry  on  human-interface  technology,  but  now 
that  has  changed.  The  gaming  industry  is  a  huge  market  that  enables  game 
developers  to  investment  huge  sums  of  money  on  research  and  development. 
According  to  Bigham,  game  developers  have  dwarfed  what  the  Department  of 
Defense  would  spend  developing  these  type  of  controllers.  Instead  of 
reinventing  the  wheel,  the  Department  of  Defense  is  adopting  this  off  the  shelf 
technology  that  millions  of  youths  are  using  on  a  daily  basis. 
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Figure  6.  U.S.  Army  Soldier  using  a  XBOX  Controller  (From  Derene,  2008) 

Raytheon  is  now  developing  unmanned  aerial  vehicles  and  improvised 
explosive  devices  disposal  robots  that  utilize  game  controllers.  They  believe  the 
use  of  game  controllers  has  the  potential  to  greatly  decrease  the  number  of 
accidents  that  occur.  Raytheon  states  that  using  off  the  shelf  game  controller 
technology  could  save  the  military  $500  million  in  about  ten  years.  In  addition  to 
advances  made  for  UAVs,  there  have  been  significant  advances  in  the  use  of 
ground  robots  as  well.  New  Scientist  magazine  reported  on  how  the  remote  is 
being  used  to  replace  traditional  joystick  controls  for  I  ED  disposal  robots.  David 
Bruemmer,  an  engineer  at  the  U.S.  Department  of  Energy’s  Idaho  National  Lab, 
says,  “The  problem  with  the  original  joypad  designed  for  the  I  ED  disposal  robot  is 
that  it  takes  a  lot  of  concentration  of  the  soldier  using  it,”  Bruemmer  also  goes  on 
to  say  that  the  game  controllers  are  more  intuitive  because  hand  movements 
transfer  directly  into  movements  of  the  robot.  This  feature  should  allow  a  more 
instinctive  approach  to  controlling  the  robot  (Flambling,  2008). 
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While  some  say  that  those  spending  time  playing  games  such  as  Halo  3 
and  Modern  Warfare  are  wasting  time,  others  believe  that  they  are  undergoing  a 
form  of  training  that  does  not  cost  anything.  When  training  to  fly  F-16s,  soldiers 
in  their  early  20s  preferred  a  game  controller  to  the  throttle  and  stick  controls.  In 
addition  to  integrating  XBOX  game  controllers  into  their  combat  systems, 
Raytheon  has  also  been  experimenting  with  using  the  Wii  controller  for  training 
that  requires  physical  movement  (Derene,  2008). 

G.  PERSONAL  COMPUTER  VIDEO  GAMES  VERSUS  CONSOLE  VIDEO 

GAMES 

1.  Background 

In  the  modern  world  of  gaming  today,  there  is  an  ongoing  debate 
regarding  which  is  better,  a  PC  or  a  video  game  console.  In  the  beginning,  there 
was  no  denying  the  advantage  of  the  PC.  As  video  games  originated  on  PCs, 
consoles  were  viewed  as  primitive  and  lacking.  As  the  console  progressed,  it 
began  to  have  more  and  more  of  the  same  features  as  the  PC.  As  time 
continues  forward,  the  debate  between  PCs  and  consoles  grows  larger.  There 
are  many  topics  that  fuel  this  debate  including  graphics,  gaming  experience, 
social  aspects,  and  the  cost  of  both  hardware  and  software. 

The  ease  and  enjoyment  of  the  gaming  or  training  experience  is  also 
important.  This  is  a  hotly  debated  question  between  gamers.  The  console 
gaming  side  believes  their  controller  interface,  reliability,  and  constancy  of 
performance  tips  the  scales  in  their  favor.  They  argue  that  the  game  software  is 
designed  to  a  closed  system  that  maximizes  the  gaming  experience.  Whereas, 
PC  games  must  perform  on  PCs  using  different  technical  configurations.  Console 
video  games'  single  configuration  results  in  much  higher  reliability  compared  to 
the  vagaries  of  PC  gaming  systems.  This  makes  it  hard  for  software  developers 
to  pinpoint  the  problem  and  fix  the  glitch.  However,  PCs  have  an  advantage  in 
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the  content  of  games.  While  consoles  still  have  a  decent  selection,  they  have 
nowhere  near  the  amount  of  games  PCs  do.  Also,  consoles  cannot  even  play  all 
of  the  games  from  their  predecessors. 

While  PCs  have  ruled  multiplayer  (two  or  more  players  playing 
simultaneously)  for  a  long  time,  they  are  now  falling  to  consoles.  Although  both 
PCs  and  consoles  allow  you  to  play  online  with  a  number  of  others,  consoles 
have  a  more  efficient  method  of  communication.  Consoles  have  built  in  voice 
chat,  whereas  PCs  have  poorly  designed  video  chat  built  into  the  game,  or  the 
need  to  open  a  third  party  program. 

The  last  issue  is  cost.  Considering  everything  needed,  a  mid  range  PC 
and  console  video  gaming  system  are  about  the  same  price.  However,  as 
technology  progresses  and  new  games  come  out,  the  PC  will  become  outdated 
faster  than  the  console.  Recently,  console  video  game  companies  readily 
admitted  that  they  lose  money  when  selling  the  baseline  console  video  games 
system.  These  systems  are  so  cutting  edge  at  their  release  date,  that  the 
research,  development,  and  manufacturing  cost  far  outweigh  the  initial 
purchasing  price.  The  gaming  companies  then  explain  it's  the  profits  from  video 
game  software  that  creates  a  significant  downstream  revenue.  At  the  moment, 
PCs  and  consoles  have  many  of  the  same  features,  and  thus  whether  PC  or 
console  is  more  enjoyable  is  simply  a  matter  of  preference. 

2.  Studies  Comparing  PC  and  Consoie  Video  Games 

There  have  been  very  few  studies  comparing  PC  based  video  games  and 
console  based  video  games  for  military  training  purposes.  A  team  of  Swedish 
scientists  from  Linkoping  University  and  Jonkoping  University  did  conduct  a  study 
in  2009  focusing  on  visual  search  strategies  using  both  PC  and  console  video 
game  platforms.  The  aim  of  the  study  was  to  compare  military  trained  personnel 
with  non-military  trained  personnel  with  respect  to  visual  fixation  techniques  to 
identify  object  of  interest  and  areas  of  interest.  One  of  the  controls  was  varying 
the  game  platform  from  PC  and  Console. 
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The  video  game  chosen  was  the  First  to  Fight™  game  (Destineer  Studios). 
The  XBOX  was  chosen  as  the  console  video  game  platform.  The  researchers 
found  very  little  difference  in  performance  between  the  PC  video  game 
configuration  versus  the  console  video  game  configuration.  At  the  end,  they  did 
state  it  seemed  that  there  was  a  non-significant  trend  toward  the  PC 
outperforming  the  console  video  game  configuration  (Falkmer,  2010). 

H.  VIRTUAL  BATTLESPACE  2™  (VBS2^“) 

Bohemia  Interactive  Virtual  Battlespace  2™  (VBS2™)  is  the  successor  of 
the  Virtual  Battlefield  Simulation  (VBS™).  It  was  developed  in  close  cooperation 
with  the  United  States  Marine  Corps  (USMC),  Australian  Defence  Force  and 
other  military  customers  of  the  original  VBS™.  VBS2™  was  officially  launched  on 
17  April  2007  and  purchased  by  the  U.S.  Army  in  February  2009. 

VBS2™  is  a  fully  interactive,  three-dimensional  training  system  providing  a 
synthetic  environment  suitable  for  a  wide  range  of  military  training  and 
experimentation  purposes.  It  offers  battlefield  simulations  and  the  ability  to 
operate  land,  sea,  and  air  vehicles.  Instructors  may  create  new  scenarios  and 
then  engage  the  simulation  from  multiple  viewpoints.  VBS2™  is  used  to  teach 
tactics,  techniques,  and  procedures  at  the  lower  tactical  level  in  multiple  types  of 
operations  (VBS2™  Manual,  2009). 

It  provides  a  high-fidelity  virtual  environment,  suitable  for  squads  and 
platoons  to  train  mission  rehearsals,  area  of  operations  (AO)  familiarization, 
convoy  operations,  react  contact  drills  (improvised  explosive  devices  (lED), 
snipers,  ambushes  and  other  insurgent  tactics),  troop  leading  procedures  and 
tactical  decision  making. 

VBS2™  is  a  fully-featured  training  tool  including  after-action  review 
capability,  high  level  architecture  (FI LA)  /  distributed  interactive  simulation  (DIS) 
compliance,  and  a  comprehensive  yet  easy-to-use  mission  editor  that  allows  any 
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imaginable  scenario  to  be  created  and  also  modified  in  real  time.  Mission  design 
simplicity  is  improved  with  a  new  “point  and  click”  interface,  which  replaces  the 
time  consuming  text  driven  editor. 

Capabilities  include,  but  are  not  limited  to,  multiple  weapon  platforms  and 
vehicles,  integrated  close  air  support  (CAS)  &  fire  support  simulation.  It  also 
provides  the  ability  to  customize  terrain  and  build  scenarios  to  meet  unit  training 
needs.  For  example,  day,  night  or  limited  visibility  situations  are  available  to 
integrate  night  vision  and  thermal  site  capability  into  customized  scenarios.  The 
ability  to  customize  training  scenarios  is  currently  unmatched  by  anything  the 
U.S.  Army  has  in  its  training  simulations  arsenal.  This  capability  is  provided  to 
VBS2™  through  the  use  of  the  commercial  gaming  engine  Real  Virtuality  2 
created  by  Bohemia  Interactive  (VBS2™  Manual,  2009). 

VBS2™  has  enjoyed  some  scientific  scrutiny,  and  this  prior  research  has 
shed  some  light  on  the  time  required  to  learn  and  use  the  simulation  as  an 
extension  of  one’s  warfighting  skills.  In  2004,  an  Australian  research  team 
conducted  a  weeklong  trial  to  determine  the  potential  utility  of  the  VBS2^''^ 
predecessor,  VBS2™.  The  trials  involved  a  group  of  participants  varying  in 
computer  and  gaming  experience  with  roughly  half  the  people  having  no  gaming 
experience.  The  study  determined  that  nearly  80  percent  of  new  users  can  attain 
individual  skill  proficiency  within  a  couple  hours.  For  higher  level  cognitive  skills, 
such  as  situational  awareness  and  team  leadership,  participants  needed  up  to 
two,  eight  hour  days  to  become  proficient  in  the  game  play  (Morrison,  Barlow, 
Bethel,  &  Clothier,  2005). 

There  are  several  gaps  in  understanding  how  to  most  effectively  use 
VBS2™  as  a  training  device  for  Soldiers.  This  thesis  attempts  to  address  some 
of  these  gaps.  Specifically,  we  investigate  the  impact  of  using  a  Sony  PS3™ 
Game  Controller  to  replace  the  standard  keyboard  and  mouse  as  an  user-input 
device.  It  also  address  the  question,  does  the  addition  of  Naturalpoint's  TrackIR 
5™  improve  the  performance  of  the  participant? 
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I. 


PLAYSTATION  3  GAME  CONTROLLER 


1.  PlayStation  (PS)  Game  Controller 

Sony's  original  PlayStation  (PS)  game  controller  was  introduced  in  1995 
and  featured  a  four  direction  D-pad,  four  action  buttons  (referred  not  by  color  or 
letter/number  like  most  pads  back  then,  but  by  four  colored  shapes  -  ,  O, 

□  ),  four  shoulder  buttons  (R1,  R2,  LI,  and  L2,  standing  for  right  and  left)  and 
start  and  select  buttons.  The  basic  design  and  layout  were  based  on  that  of 
Nintendo's  Super  Nintendo  Entertainment  System  (SNES)  controller,  as  the 
PlayStation  was  originally  developed  for  the  SNES,  before  becoming  a  console  in 
its  own  right. 

2.  PlayStation  2  (PS2)  Game  Controller 

In  2000,  Sony  released  PS2  with  the  introduction  of  Sony's  PS2 
DualShock  game  controller.  This  device  could  vibrate  to  provide  the  player  with 
tactile  feedback,  cosmetically  changed  the  handles,  analog  sticks  and  “L2  and 
R2”  buttons,  and  added  “L3  and  R3”  buttons,  which  were  incorporated  into  the 
sticks  themselves  (accessed  by  pushing  down  on  the  stick).  Its  popularity 
resulted  in  the  DualShock  becoming  the  de  facto  standard  that  all  other  console 
game  controller  were  compared.  Other  minor  modifications  made  include  the 
change  of  cable  and  end  connector  color  from  grey  to  black,  a  slight  squaring  of 
the  connector  and  “DUALSHOCK  2”  printed  in  blue  on  the  top  of  the  controller 
next  to  where  the  cable  enters. 

3.  PlayStation  3  (PS3™)  Game  Controller 

Initially,  the  conceptual  controller  for  the  PlayStation  3  was  similar  to  its 
Dual  Shock;  however,  it  was  much  more  curved  in  shape  than  these  controllers, 
with  an  appearance  similar  to  that  of  a  banana  or  boomerang.  This  odd  shape 
has  often  been  the  subject  of  much  ridicule  and  was  abandoned  in  2006. 
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The  PlayStation  3's  Dual  Shock  3  wireless  controller  includes  both 
vibration  function  and  the  motion-sensing  functionality.  The  DualShock  3  is 
cosmetically  similar  to  the  DualShock  2,  but  has  several  new  features  that 
distinguish  it.  It  is  a  wireless  controller  and  features  the  addition  of  tilt-sensor  and 
linear  accelerometer  technology,  as  well  as  larger  'trigger-like'  L2  and  R2 
buttons.  It  is  charged  by  way  of  a  mini-USB  port  situated  where  the  cable  on  its 
predecessors  left  the  controller,  which  can  also  be  used  to  allow  wired  play. 
However,  the  new  controller  lacks  the  rumble  capability  featured  in  the  preceding 
DualShock  2  controllers  due  to  patent  infringement  legal  issues  with  the  haptic- 
feedback  technology  (see  Figure  5). 

Having  settled  the  legal  issue  in  late  2007,  Sony  relaunched  the 
DualShock  3.  The  controller  is  the  same  as  the  previous  DualShock  3  in  almost 
every  way  but  differs  in  that  the  blue  lettering  next  to  the  USB  port  has  been 
changed  to  DualShock  3,  it  contains  the  aforementioned  vibration  motors,  which 
also  give  the  controller  more  weight.  This  study  will  use  the  Sony  PlayStation  3 
game  controller  as  the  alternative  user-input  device. 

J.  NATURALPOINTTRACKIR5™ 

1.  Introduction 

TrackIR  5™  is  an  optical  motion  tracking  game  controller  for  Microsoft 
Windows,  created  by  Naturalpoint  Inc.  that  tracks  head  motions  with  up  to  six 
degrees  of  freedom  (6DOF),  allowing  hands  free  view  control  for  improved  game 
immersiveness  and  situational  awareness.  Head  position  and  orientation  are 
measured  by  a  purpose-built  video  camera,  mounted  on  top  of  the  user's 
monitor,  which  observes  invisible  infrared  (IR)  light  (hence  the  name)  reflected  or 
emitted  by  markers  on  a  rigid  model  worn  by  the  user.  TrackIR  ™  software  is 
used  to  access  and  control  the  camera,  as  well  as  adjust  tracking  and  manage 
game  profiles.  The  ratio  of  actual  head  movement  to  virtual  head  movement  can 
be  changed,  allowing  the  virtual  head  to  turn  180  degrees  whilst  the  user  only 
turns  his  head  a  little  bit  and  still  looks  at  the  monitor  (NaturalPoint,  2010). 
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The  actual  kit  consists  of  two  main  pieces;  a  universal  serial  bus  (USB) 
infrared  (IR)  camera  that  sits  on  top  of  your  main  monitor,  where  it's  held  by  a 
simple  adjustable  mount.  The  second  is  the  TrackClip™,  a  passive  clip  with 
three  reflective  areas  that  hooks  onto  the  brim  of  a  baseball  cap.  The  physical  set 
up  is  simple  and  installing  the  software  is  equally  easy. 


Figure  7.  TrackIR  5  Infrared  Camera  (left)  and  TrackClip  (right)  (From 

NaturalPoint,  2010) 


2.  Technical  Details 

TrackIR  5™  cameras  have  a  monochrome  sensor  with  an  on-board 
programmable  logic  device  that  pre-processes  grayscale  video  into  a  threshold 
binary  video.  This  video  is  sent  to  the  computer  via  USB  and  further  processed 
by  TrackIR  software  to  locate  markers  and  from  this,  estimate  head  position  and 
orientation,  all  using  only  a  small  amount  of  bandwidth  and  CPU  (NaturalPoint, 
2010). 

Now  in  its  fifth  iteration,  TrackIR™  is  a  system  designed  to  turn  real  head 
movements  into  virtual  head  movements.  It  works  with  a  number  of  games  that 
provide  a  virtual  cockpit  view,  and  it  effectively  uses  your  head  as  a  kind  of 
mouse-look.  Turn  your  head  left,  and  your  view  in-game  rotates  left.  Turn  right, 
and  your  view  rotates  right.  Tilts  of  the  head  up  and  down  work  along  similar 
lines.  Flowever,  TrackIR  5™  goes  further.  In  games  with  full  support  for  all  six 
degrees  of  movement,  you  can  raise  your  head  up  and  lower  it  down,  lean  left 

and  right  or  move  backwards  and  forwards,  and  your  view  adjusts  accordingly. 
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And  as  the  system  exaggerates  small  head  movements  into  big  ones  in-game, 
you  never  have  to  turn  so  far  that  you  can't  see  the  screen.  It  might  not  be  quite 
as  convincing  as  being  surrounded  by  a  bank  of  monitors  in  a  serious, 
commercial  simulation,  but  it  should  be  more  immersive  than  a  single  static  video 
monitor. 

TrackIR  5™  has  promoted  and  established  head-tracking  in  PC  games, 
achieving  support  from  many  developers,  particularly  those  involved  in  the 
simulation  genre.  In  the  process,  its  interface  has  become  the  de  facto  standard 
for  head-tracking  game  view  control,  although  other  commercial  and  non¬ 
commercial  solutions  exist  (Wallis,  2006).  This  study  will  use  the  NaturalPoint 
TrackIR  5™  as  an  additional  user-input  device. 


Figure  8.  Participants  using  the  NaturalPoint  TrackIR  5™ 
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III.  EXPERIMENT  AND  SCENARIO  DEVELOPMENT 

METHODOLOGY 


A.  INTRODUCTION 

In  order  to  determine  the  correct  configuration  of  VBS2™  peripheral 
interfaces,  we  must  define  the  problem  and  the  game  skills  required  to  conduct 
military  training  using  VBS2  ™.  Once  the  problem  and  required  skills  are 
defined,  the  scenario  can  then  be  created.  Then  we  will  determine  how  to 
measure  participants’  performance.  We  defined  the  problem  as,  which  VBS2™ 
interface  configuration  provided  the  most  intuitive  setup  resulting  in  minimum 
initial  training  time  and  maximum  time  spent  focused  on  military  training  instead 
of  learning  a  game. 

The  experiment  and  scenario  development  was  broken  down  into  three 
phases.  The  three  phases  were  as  follows: 

Phase  I:  Define  the  scenario  and  how  to  measure  performance 

Phase  II:  Design  and  develop  the  scenario  in  VBS2™ 

Phase  III:  Design  data  collection  systems 

The  scenario  and  data  design  and  development  was  co-conducted  with 
Major  Benjamin  Brown  of  the  United  States  Marine  Corps.  Major  Brown  was 
MOVES  student  attending  NPS  during  this  critical  phase.  Major  Brown  and  I 
were  classmate  in  three  courses  and  teamed  together  to  complete  course 
requirements  that  directly  related  to  the  development  of  my  thesis  experiment, 
specifically,  scenario  development  and  data  collection  systems.  In  this  chapter, 
the  term  'we'  refers  to  Dr.  Bill  Becker,  Dr.  Quinn  Kennedy,  Major  Ben  Brown,  and 
me. 
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B.  PHASE  I:  DEFINE  THE  SCENARIO  AND  HOW  TO  MEASURE 

PERFORMANCE 

1.  What  Skills  are  Required  for  Conducting  Military  Training 
Using  Video  Games 

The  first  step  in  defining  the  scenario  is  to  determine  the  most  important 
basic  gaming  skills  a  soldiers  must  possess  in  order  to  use  VBS2™  as  an 
effective  virtual  desktop  trainer.  These  are  the  skills  that  a  soldier  must  already 
possess  or  can  quickly  learn  during  a  familiarization  exercise. 

Each  skill  chosen  would  have  to  meet  three  criteria.  First,  the  skill  must  be 
required  to  effectively  participate  in  a  virtual  military  training  event.  Second,  it 
must  be  measurable.  Third,  the  participant's  ability  to  execute  the  skill  should 
reflect  the  participants’  natural  ability,  gaming  experience,  and  interface 
configuration. 

Shoot,  move,  and  communicate.  These  three  words  encompass  a 
Soldier's  most  basic  training  directive.  If  Soldiers  can  effectively  do  all  three  of 
these  things,  they  will  be  successful  in  any  tactical  mission  they  encounter. 
Knowing  how  to  shoot,  move  and  communicate  is  so  vital  to  the  success  of  a 
mission,  tactical  training  often  focuses  on  these  basics  skills.  "These  are  the 
fundamental  basics  that  will  get  the  Soldiers  to  more  complex  maneuvers” 
(Deweese,  2010). 

The  two  skills  we  decided  to  focus  on  were  shooting  and  moving.  In  the 
early  stages  of  this  project,  we  decided  to  focus  on  the  individual,  and  exclude 
any  collective  training  tasks.  By  focusing  on  individual  tasks,  we  eliminated  the 
variability  introduced  when  communicating  in  a  virtual  environment. 
Furthermore,  the  two  user-input  devices  we  examine  in  this  project  have  no 
impact  on  Soldier  communication  skills.  We  agreed  to  exclude  communication 
because  it  was  outside  the  scope  of  our  project.  As  a  side  note,  during  our 
background  research,  we  discovered  that  some  gamers  used  the  head-tracking 
device  to  nod  their  head.  This  head  nod  was  used  to  signal  an  affirmative  or 
negative  response. 
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The  third  skill  we  focused  on  was  the  ability  to  observe  the  immediate 
environment.  This  skill  was  driven  by  our  desire  to  examine  the  usefulness  of  a 
head-tracking  device  during  military  training  events.  Our  original  idea  was  to 
incorporate  situational  awareness  as  an  essential  skill.  We  conducted  an 
interview  with  the  Naturalpoint  TrackIRS  product  manager  at  the 
Interservice/Industry  Training,  Simulation,  and  Education  Conference  (l/TSEC) 
and  were  told  that  gamers  continually  cited  improved  situational  awareness  as 
the  main  reason  they  purchase  and  recommend  using  a  Naturalpoint  TrackIR  5™ 
head-tracking  device.  The  challenge  we  encountered  was  defining  situational 
awareness  and  then  developing  metrics.  After  debating  the  issue  for  quite  a 
while,  we  decided  to  scope  down  the  situational  awareness  skill  to  a  much  more 
definable  and  measurable  ability:  to  observe  the  immediate  environment. 

After  this  analysis,  we  decided  to  focus  on  the  following  five  skills  during 
our  scenario  design. 

•  Individual  movement 

•  Individual  shooting  a  rifle 

•  Mounted  HMMWV  movement 

•  Mounted  HMMWV  shooting  a  M2  0.50  caliber  machine  gun 

•  Ability  to  observe  the  immediate  environment 

As  described  below  (Table  1),  each  skill  was  measured  by  time  to 
complete  and  by  accuracy  level. 

2.  Define  the  Movement  Technique:  Dismounted,  Mounted,  or 

Both? 

Full  spectrum  operations  today  dictate  that  soldiers  have  the  ability  to 
wage  war  mounted  and  dismounted.  In  the  1980s  and  90s,  combat  Soldiers 
were  either  considered  'heavy',  mounted  on  vehicles,  or  'light',  dismounted  and 
inserted  via  air  platforms.  The  heavy  community  often  integrated  virtual  training 
simulations  into  their  unit  training  plans  due  to  simulations  huge  cost  saving.  In 
the  light  community,  training  exercises  are  much  less  expensive,  thus  the  cost 
saving  is  much  less  significant.  This  resulted  in  the  heavy  community  embracing 
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simulations  training  technologies  far  more  than  the  light  community.  This 
difference  in  cost  savings  has  changed  in  recent  years  as  wartime  commanders 
look  for  every  advantage  in  training  their  units  for  combat  operations.  This  study 
will  evaluate  participants  on  both  dismounted  and  mounted  movement 
techniques. 

C.  PHASE  II:  DESIGN  AND  DEVELOP  THE  SCENARIO  IN  VBS2™ 

Once  we  had  defined  the  problem,  identified  the  critical  skills  sets, 
established  measures  of  the  chosen  skill  sets,  and  decided  on  the  conceptual 
design  of  the  scenario,  we  built  the  scenario  into  VBS2™.  Special  care  was 
given  to  building  the  scenario.  The  scenario  must  first  focus  on  the  user  skill 
gained  during  the  initial  training  session.  The  scenario  focused  on  the  five  skill 
listed  above  (individual  dismounted  movement  (walking),  mounted  movement 
(driving),  firing  individual  personnel  weapon  (M4  Rifle),  firing  crew  served 
weapon  (M2  .50  Cal  Machine  Gun),  and  situational  awareness).  The  scenario 
must  also  be  complex  enough  to  cause  participants'  results  to  be  reflective  of 
their  natural  ability,  gaming  experience,  and  type  of  configuration.  The  distance 
and  dispersion  of  the  targets  on  the  range  must  be  varied  from  easy  to  hard. 
This  manipulation  should  provide  a  challenge  to  all  participants  regardless  of 
their  gaming  experience  or  VBS2™  configuration  type,  enabling  us  to  capture 
real  individual  difference.  Finally,  the  scenario  must  be  simple  enough  to  avoid 
introducing  unnecessary  sources  of  variation  into  the  data.  For  example,  adding 
agents  representing  civilians  and  insurgents  might  add  reality  and  challenge  but 
would  likely  introduce  considerable  noise  to  the  data,  so  they  were  used 
sparingly.  Finally,  we  avoided  building  tactics  into  the  scenario  to  prevent  prior 
knowledge  of  tactics  from  becoming  an  independent  variable. 

The  course  consists  of  a  mounted  vehicle  movement  portion  followed  by 
an  individual  dismounted  movement  portion  (see  Figure  9).  The  participant  starts 
at  a  designated  point,  drives  the  vehicle  through  a  serpentine  course,  and  stops 
at  battle  position  1  of  the  mounted  shooting  range.  There,  the  participant 
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switches  from  the  driver’s  seat  to  the  gunner’s  station  and  engages  three  truck 
targets.  Afterward,  the  participant  returns  to  the  driver’s  seat  and  continues  on 
the  driving  course.  The  participant  must  negotiate  a  90  degree  left  turn  with 
buildings  on  both  side  of  the  road.  Additionally,  two  civilian  vehicles  approach  in 
this  area,  and  the  participant  must  negotiate  around  the  traffic.  Once  through  the 
built  up  area,  the  participant  stops  at  battle  position  2  of  the  mounted  shooting 
range  and  engages  an  additional  three  truck  targets.  Once  the  engagement  is 
complete,  the  participant  continues  on  the  movement  course,  negotiating  two 
more  civilian  vehicles  worth  of  traffic.  At  the  end  of  the  course,  the  participant 
combat  parks  the  vehicle  by  backing  it  between  traffic  barriers  and  cones. 
Measures  of  performance  include  the  following:  number  of  vehicle  collisions  with 
scenery  objects  including  traffic,  number  of  times  all  four  wheels  of  the  vehicle 
went  off  the  road,  number  of  targets  hit,  number  of  rounds  expended,  and  time 
for  each  portion  of  the  course. 
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Figure  9.  A  simplified  schematic  of  the  course  scenario  developed  for  the 

project. 


The  individual  movement  portion  of  the  course  starts  when  the  participant 
dismounts  from  the  parked  vehicle.  The  participant  walks  into  an  arrangement  of 
buildings  that  includes  closed  alleys.  Each  stall  features  either  an  enemy  or 
friendly  target.  The  twelve  alleys  are  paired  into  six  groups.  In  each  pair,  the 
alleys  are  on  opposite  sides  of  the  road  so  that  the  participant  must  look  fully  to 
the  left  and  to  the  right  to  determine  which  target  to  engage.  Once  participants 
completes  the  shoot  /  no  shoot  course,  they  negotiate  the  ramp  weave  course 
where  they  walk  across  elevated  ramps  arranged  at  different  angles.  Next, 
participants  negotiate  two  weave  courses.  One  has  walls  high  enough  that 
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participants  cannot  see  over  the  top  and  have  no  awareness  of  their 
surroundings  or  direction.  The  other  has  low  walls  so  that  participants  can  see 
over  the  top  of  the  whole  course.  At  the  end  of  the  low  weave,  the  participants 
low  crawl  through  a  tunnel,  stand  and  run  to  the  individual  shooting  range. 
There,  participants  engage  personnel  targets.  Performance  measures  for  the 
individual  portion  include:  number  of  falls  off  the  ramps,  number  of  enemy  targets 
hit,  number  of  friendly  targets  hit,  number  of  rounds  expended,  and  time  for  each 
portion  of  the  course.  Table  1  summarizes  the  full  interface  evaluation  course. 


Station 

Participant  Action 

Performance  Measure 

Mounted  weave 

Drive  around  barricades  as  quickly  as 
possible  without  wrecking  the  vehicle 

•  Time 

•  Numberof  vehicle  crashes 

Mounted  shooting 
range,  Station  1 

Use  the  HMMWV  M2  to  destroy  3 
enemy  trucks 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Mounted  shooting 
range,  Station  2 

Use  the  HMMWV  M2  to  destroy  3 
enemy  trucks 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Mounted  movement 

Move  from  mounted  movement 
course  start  to  stop,  with  time  at 
mounted  shooting  range  stations 
subtracted 

•  Time 

•  Number  scenery  objects  hit 

•  Number  times  vehicle  off  road 

Ability  to  observe 

Combat  park  vehicle 

Conduct  shoot  /  no  shoot  alleys 

•  Time 

•  Number  targets  hit 

•  Number  friendlies  hit 

Individual  movement 

Move  through  ramp  weave,  high 
weave,  and  low  weave 

•  Time 

•  Number  falls  off  ramp 

Individual  shooting 
range 

Use  the  rifle  to  shoot  6  short  range 
personnel  targets  and  3  long  range 
popup  targets 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Table  1 .  Interface  evaluation  course  station  guide 


To  statistically  compare  the  performance  of  various  participants,  the 
project  proposes  a  scoring  system  that  can  be  used  to  combine  the  various 
performance  measures  into  quantifiable  measures.  Generally,  the  scoring 
system  is  divided  into  time  portions  and  accuracy  portions  for  each  skill  set. 
These  scores  will  be  normalized  and  used  as  the  basis  for  analysis. 
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D.  PHASE  III:  DESIGN  DATA  COLLECTION  SYSTEMS 

1.  Introduction 

We  planned  to  gather  our  experimental  data  from  three  sources.  The  first 
was  the  initial  survey  of  participants  to  establish  a  personal  profile  baseline.  The 
second  source  came  from  using  the  After  Action  Review  feature  in  VBS2™  to 
evaluate  the  performance  of  the  participants  during  the  evaluated  portion  to  the 
exercise.  For  example,  we  wanted  to  establish  a  scoring  system  that  evaluates 
how  long  it  takes  participants  to  complete  an  obstacle  course,  how  many  times 
their  vehicle  leaves  the  road,  and  his  performance  on  a  weapons  firing  range  with 
regards  to  time  and  accuracy.  The  final  source  was  a  second  survey  evaluating 
the  training  exercise. 

2.  Surveys 

Consistency  plays  a  key  role  in  the  validity  of  subjective  research 
assessments.  One  technique  for  achieving  consistency  involves  automating  the 
surveys.  While  automation  introduces  the  potential  for  system  error,  it  eliminates 
the  possibility  of  human  error.  A  well  tuned  automate  survey  system  with  low 
system  error  risk  can  dramatically  improve  the  consistency  of  results,  thereby 
contributing  to  a  more  solid  analysis  and  better  supported  conclusion. 

Previous  MOVES  experiments  using  Soldiers  as  participants  and  surveys 
to  gather  additional  experimental  data  have  fallen  short  of  researcher 
expectations.  The  single  biggest  problem  was  with  paper  based  surveys  where 
participants  failure  to  answer  questions  in  a  meaningful  way.  It  was  highly 
recommended  that  we  avoid  paper  surveys  and  automate  the  survey  process 
because  it  greatly  improves  the  quality  of  our  survey  data. 

Automating  the  survey  process  serves  several  purposes.  First,  if  forces 
participants  to  answer  the  question,  even  if  it  is  only  “I  do  not  wish  to  answer  this 
question”.  Second,  it  provides  the  ability  to  categorize  answers  and  conduct 
error  checking.  For  example,  if  asked  “how  many  years  have  you  played  video 
games”  an  answer  of  'IV  would  send  the  user  into  a  loop  until  he  entered 
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correctly  entered  '17'.  Finally,  an  automated  survey  could  produce  a  text  file  that 
could  easily  be  imported  into  our  statistical  software  package. 

We  used  three  surveys  to  gather  demographic  data  and  preference  data. 
The  first  survey  was  a  pre-exercise  survey  designed  to  gather  demographic  data 
on  the  participants.  The  second  was  a  post-exercise  survey  completed  after  the 
first  run  to  gather  participant  preference  data  on  the  user  initial  input 
configuration.  The  third  survey  was  a  post-exercise  survey  completed  after  the 
final  run.  It  asked  the  same  questions  as  the  first  post-exercise  survey  but  added 
some  additional  questions  focused  on  identifying  participant  preference  between 
the  two  configurations. 

3.  Automated  Scoring  System 

a.  Introduction 

As  mentioned  in  the  previous  section,  consistency  plays  an 
important  role  in  the  validity  of  objective  research  assessments.  A  second 
technique  for  achieving  consistency  involves  automating  the  scoring  process  as 
much  as  possible.  Our  method  used  Java  code  to  read  VBS2™  data  packets  to 
automate  the  scoring  process. 

Calytrix  Technologies  worked  with  Bohemia  Interactive  to  develop 
supplemental  software  for  VBS2™  to  communicate  with  other  military  simulation 
products.  The  United  States  military  has  adopted  two  protocol  standards  for 
simulations:  Distributed  Interactive  Simulation  (DIS)  and  High  Level  Architecture 
(HLA).  The  Calytrix  product  LVC  Game  supports  both  protocols,  enabling 
VBS2™  to  communicate  via  DIS  or  HLA. 

DIS  uses  User  Datagram  Protocol  (UDP)  to  facilitate 
communications.  UDP  involves  sending  packets  across  a  network  in  an 
unsupervised  fashion.  UDP  packets  may  arrive  in  duplicate,  may  not  arrive  at  all, 
or  may  arrive  out  of  order.  However,  because  UDP  packets  flow  as  quickly  as 
sender  and  receiver  can  operate,  it  is  ideal  for  the  world  of  game  applications 
where  activity  is  constant.  While  DIS  is  a  complex  domain  in  military  simulation, 
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the  concept  is  quite  simple.  DIS  provides  a  coding  system  for  UDP  packets.  The 
Simulation  Interoperability  Standards  Organization  (SISO)  maintains  a  codebook 
for  DIS.  Assuming  everyone  involved  adheres  to  the  SISO  standards,  any  DIS 
user  can  communicate  to  another  across  any  network  regardless  of  platform  or 
gaming  application.  DIS  has  been  used  successfully  for  more  than  a  decade, 
and  international  partners  have  adopted  the  standard  in  order  to  use  simulation 
in  joint  training.  The  concept  of  DIS  is  alive  and  well,  providing  a  dependable 
means  of  communicating  with  VBS2™. 

b.  Method 

This  project  sought  to  use  Java  code  to  read  DIS  packets  from 
VBS2™  and  interpret  the  information  into  viable  information  for  performance 
measures.  We  used  an  open  source  library  called  Open-DIS,  developed  and 
maintained  at  the  Naval  Postgraduate  School,  to  process  DIS  PDU’s  into  Java 
objects  for  easy  manipulation.  The  project  started  by  reading  DIS  packets  from 
the  game  in  a  simple  scenario.  Once  the  general  content  of  the  VBS2™  DIS 
packets  was  evaluated,  ad  hoc  methods  of  using  VBS2™  scenario  editing  were 
developed  in  order  to  achieve  the  goals  of  the  various  performance  measures. 
Initial  project  analysis  dictated  the  ability  to  do  the  following  tasks: 

•  Detect  target  hits; 

•  Detect  number  of  rounds  fired; 

•  Record  times  of  various  events; 

•  Detect  collisions. 

c.  Determine  What  Information  VBS2™  Packets  Could 
Provide 

DIS  data  packets  are  called  Protocol  Data  Units  (PDUs).  DIS  offers 
a  variety  of  PDU  types,  and  the  PDU  type  is  the  fundamental  component  of 
message  decoding.  Depending  on  the  PDU  type,  all  other  information  in  the 
packet  has  a  certain  distinct  meaning.  Up  front,  we  identified  four  PDU  types  of 
potential  interest  to  this  project:  Entity  State,  Fire,  Detonation,  and  Collision.  We 
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used  a  simple  scenario  to  determine  which  PDUs  we  could  detect  from  VBS2™. 
After  maneuvering  and  shooting  both  mounted  and  dismounted,  we  determined 
that  we  could  detect  entity  state,  fire,  and  detonation  PDUs.  We  could  not  detect 
collision  PDUs.  As  a  result  we  determined  that  detecting  vehicle  collisions  with 
scenery  objects  or  traffic  and  individual  falls  off  the  ramps  would  not  be  possible. 
Additionally,  we  determined  that  none  of  the  PDUs  could  be  effectively  used  to 
determine  if  a  vehicle  ran  all  four  wheels  off  the  road.  Because  these  three 
measures  could  be  detected  in  after  action  review  footage,  we  determined  that 
they  would  have  to  be  human  evaluated. 

In  our  evaluation  of  the  PDUs,  we  determined  that  the  following 
information  could  be  exploited  for  our  project.  All  PDUs  included  a  time  stamp 
that  made  time  determinations  relatively  straightforward.  Entity  state  PDUs 
included  a  variety  of  coded  information  such  as  platform  name  and  country  of 
origin.  Fire  PDUs  allowed  access  to  this  entity  state  information  for  the  shooter, 
while  detonation  PDUs  allowed  access  to  entity  state  information  for  the  target. 
Each  game  entity  has  a  unique  identification  number  called  the  Entity 
Identification  (ID).  The  Entity  ID  is  a  key  piece  of  data,  but  it  is  randomly 
assigned  for  each  scenario  run,  so  the  tag  cannot  be  hard  coded. 

We  were  able  to  use  a  combination  of  the  entity  state,  fire,  and 
detonation  PDUs  to  accomplish  all  of  the  performance  objective  goals  related  to 
shooting.  However,  getting  the  exact  information  required  some  ingenuity  and 
imagination. 


d.  Recording  Shooting  information 

We  could  easily  record  the  number  of  rifle  rounds  fired  because 
only  the  rifle  produced  fire  PDUs.  Recording  the  number  of  expended  rifle 
rounds  was  simply  a  matter  of  counting  fire  PDUs.  Recording  the  number  of 
machine  gun  rounds  expended  was  a  little  less  straightforward  because  the 
machine  gun  only  produced  detonation  PDUs.  Fortunately,  every  round 
produced  a  detonation  PDU,  so  the  process  resulted  in  simply  counting  all 
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detonation  PDUs,  determining  which  denotation  PDUs  had  been  caused  by  a 
rifle,  and  subtracting  to  yield  detonation  PDUs  caused  by  a  machine  gun. 


Recording  target  hits  was  a  little  less  evident.  The  process  was 
complicated  by  the  fact  that  we  sought  to  be  able  to  distinguish  between  vehicle, 
personnel,  popup,  and  friendly  targets.  An  additional  challenge  arose  because  a 
training  portion  of  the  scenario  that  was  not  evaluated  at  all  included  similar 
vehicle  and  popup  targets.  We  also  wanted  to  record  target  hits  only  once.  That 
is,  the  code  needed  to  discern  when  a  target  had  been  hit  and  never  record  hits 
on  that  target  again.  We  detected  target  hits  through  the  detonation  PDUs. 
Detonation  PDUs  provided  both  Entity  ID  and  entity  state  information. 

The  coding  and  registration  process  adequately  facilitated 
recording  target  hits.  A  detonation  PDU  provided  Entity  ID  and  entity  state.  If  the 
entity  state  had  a  country  code  matching  the  target  range,  we  looped  through  the 
target  array  looking  for  the  Entity  ID.  If  a  match  was  achieved,  we  checked  to 
see  if  the  target  had  already  been  recorded  as  a  hit.  If  it  was  still  coded  zero,  it 
was  coded  as  hit  with  a  one,  and  recorded  the  appropriate  hit.  We  used  unique 
country  codes  to  distinguish  between  vehicle,  personnel,  popup,  and  friendly 
targets  so  that  we  could  record  hits  for  each  group  separately. 

e.  Recording  Time  information 

Recording  the  time  of  individual  target  hits  was  quite 
straightforward.  Using  the  target  information  discussed  above,  we  knew  the 
detonation  PDU  that  indicated  a  target  hit.  Recording  the  target  hit  was  reduced 
to  simply  recording  the  time  stamp  from  that  detonation  PDU. 

Determining  the  time  that  the  participant  started  each  specific 
course  station  was  much  less  to  the  point.  The  VBS2™  trigger  presented  the 
obvious  opportunity  to  record  such  information.  The  trigger  allows  the  editor  to 
supply  any  viable  VBS2™  command  through  the  game’s  scripting  language.  Use 
of  triggers  can  be  extremely  simple;  for  example,  the  presence  of  a  friendly  entity 
can  set  off  the  trigger.  Unfortunately,  the  trigger  itself  does  not  send  a  PDU  of 
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any  sort.  Thus,  we  had  to  develop  a  way  to  get  a  trigger  to  send  a  PDU.  We 
developed  a  list  of  eleven  triggers  (listed  in  Table  2)  that  would  provide  the 
necessary  timing  information. 

We  decided  to  use  the  fire  PDU  as  a  mechanism  for  passing  trigger 
information.  We  created  a  set  of  unique  individual  characters.  We  gave  each 
unique  character  type  an  individual  country  code  between  20  and  45.  We  gave 
each  character  a  unique  name,  calling  them  Triggerl  through  Trigger11.  In  the 
initialization  phase  of  the  program,  we  registered  the  triggers  in  the  same  way  we 
registered  targets  as  described  above.  While  watching  entity  state  PDUs,  we 
looked  for  entities  that  matched  the  trigger  country  code  range  and  recorded  the 
entity  ID  and  country  code  in  an  array  if  it  had  not  already  been  recorded.  We 
could  then  place  triggers  at  the  appropriate  points  in  the  scenario  to  note  station 
transition  times.  We  used  the  scripting  language  in  each  trigger  to  make  a 
unique  trigger  character  fire.  This  scripting  code  required  the  name  of  the 
character  and  the  nomenclature  of  the  character’s  weapon.  By  matching  entity 
IDs,  we  could  use  the  fire  PDUs  to  determine  the  unique  country  code  of  the 
firing  individual.  We  could  then  match  this  country  code  to  the  specific  trigger 
and  record  exactly  where  the  participant  was.  As  in  other  cases,  we  used  the  fire 
PDUs  time  stamp  to  record  when  the  participant  tripped  the  trigger.  Appendix  A 
lists  all  the  coding  information  used  for  these  triggers. 
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1 

Enter  mounted  movement  course  /  course  start 

2 

Mounted  weave  start 

3 

Mounted  weave  end 

4 

Enter  mounted  shooting  range,  Station  1 

5 

Exit  mounted  shooting  range,  Station  1 

6 

Enter  mounted  shooting  range.  Station  1 

7 

Exit  mounted  shooting  range.  Station  2 

8 

Exit  mounted  movement  course 

9 

Exit  shoot  /  no  shoot  stalls 

10 

Enter  individual  shooting  range 

11 

Exit  individual  shooting  range  /  course  end 

Table  2. 

Triggers  used  for  timing  calculations 

f.  Implemented 

The  interface  scoring  system  implemented  in  Java  using  Open-DIS 
adequately  recorded  objective  performance  information  for  all  time  and  shooting 
criteria.  We  could  find  no  way  to  use  DIS  to  record  vehicle  collisions  with 
scenery  objects,  vehicle  deviations  from  the  road,  or  individual  falls  off  the  ramps. 
Table  3  summarizes  the  performance  measures  recorded  by  the  interface 
scoring  system. 


42 


Station 

Performance  Measure 

Recording  Method 

Mounted  weave 

•  Time 

Time  stamp  from  trigger  fire  PDU 

Mounted  shooting 
range,  Station  1 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Time  stamp  from  tgt  detonation  PDU 
Detonation  PDU  entity  information 
Detonation  PDU  count 

Mounted  shooting 
range,  Station  2 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Time  stamp  from  tgt  detonation  PDU 
Detonation  PDU  entity  information 
Detonation  PDU  count 

Mounted  movement 

•  Time 

•  Number  scenery  objects  hit 

•  Number  times  vehicie  off  road 

Time  stamp  from  trigger  fire  PDU 

Not  recorded 

Not  recorded 

Situationai 

awareness 

•  Time 

•  Number  targets  hit 

•  Number  friendiies  hit 

Time  stamp  from  trigger  fire  PDU 

Fire  PDU  target  information 

Fire  PDU  target  information 

individuai  movement 

•  Time 

•  Number  faiis  off  ramp 

Time  stamp  from  trigger  fire  PDU 

Not  recorded 

individuai  shooting 
range 

•  Time 

•  Number  targets  hit 

•  Number  rounds  expended 

Time  stamp  from  tgt  detonation  PDU 
Detonation  PDU  entity  information 

Fire  PDU  count 

Table  3.  Interface  scoring  system  performance  measure  summary 


4.  Manual  Collection  System 

As  previously,  mentioned  the  automated  scoring  application  could  not 
collect  all  the  scores  to  adequately  evaluate  participant  performance. 
Specifically,  we  could  not  detect  collision  PDUs.  As  a  result,  we  determined  that 
detecting  vehicle  collisions  with  scenery  objects  or  traffic  and  individual  falls  off 
the  ramps  would  not  be  possible.  Additionally,  we  determined  that  none  of  the 
PDUs  could  be  effectively  used  to  determine  whether  a  vehicle  ran  all  four 
wheels  off  the  road.  Finally,  the  vehicle  parking  event  scoring  was  also  outside 
the  capability  of  our  automated  scoring  application.  Therefore,  we  determined 
that  they  would  have  to  be  evaluated  by  someone  watching  the  after  action 
review  video. 

This  evaluation  was  conducted  by  a  research  assistant.  He  was  trained 
on  the  basic  capabilities  of  VBS2™  and  how  to  evaluate  participant  runs  and 
record  results.  His  deliverable  was  a  spreadsheet  cataloging  the  participant 
number  and  corresponding  event  scores. 
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The  research  assistant  was  provided  a  VBS2™  laptop  computer  system 
and  a  portable  hard  drive  containing  all  the  after  action  review  video  files  for  each 
participant.  There  was  no  personally  identifiable  information  (Pll)  on  the  portable 
hard  drive.  He  would  move  the  AAR  video  files  from  the  portable  hard  drive  into 
a  folder  that  could  be  accessed  by  the  VBS2™  and  then  load  up  VBS2™.  Then 
he  would  follow  the  AAR  Evaluation  Instructions  found  in  Appendix  E.  When  he 
completed  the  evaluation  of  all  participant  AAR  he  provided  us  a  detailed 
spreadsheet  with  all  the  scoring  data  required  to  conduct  our  statistical  analysis. 
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IV.  PILOT  STUDY 


A.  INTRODUCTION 

1.  Background 

A  pilot  study  was  conducted  to  gather  preliminary  data  addressing  the 
main  research  question  and  to  refine  procedures  and  logic  of  the  main  study.  In 
this  pilot  study,  performance  with  the  standard  keyboard  and  mouse  was 
compared  to  performance  with  the  Play  Station  3  (PS3)  game  controller. 
Subjective  preference  also  was  assessed. 

2.  Research  Questions 

This  pilot  study  addresses  these  issues  by  comparing  the  standard 
keyboard  and  mouse  interface  to  the  game  controller  interface  on  user 
preference  and  objective  performance  measures.  It  specifically  address  the 
following  questions: 

1 .  Do  user-input  devices  impact  military  training  performance  when  used  with 

a  desktop  virtual  training  simulation? 

2.  Do  soldiers  prefer  one  user-input  device  over  another  when  using  a 

desktop  virtual  training  simulation? 

3.  Hypothesis 

The  addition  of  a  commercial  game  controller  device  to  virtual  desktop 
simulations  will  improve  performance  in  a  virtual  training  simulation  and  be 
preferred  over  the  standard  keyboard  and  mouse. 

4.  Scope 

As  this  was  the  pilot  study  for  the  main  study,  we  needed  to  scope  the 
project  to  focus  on  the  most  problematic  aspects  of  the  experiment:  testing  our 
methods,  procedures,  and  logic  by  comparing  the  standard  keyboard  and  mouse 
configuration  and  the  PS3  game  controller.  Specifically,  it  focuses  on  individual 
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mounted  and  dismounted  movement  within  the  VBS2™  virtual  training 
environment.  It  also  focuses  on  the  ability  to  accurately  identify  and  engage 
targets  with  personal  and  crew  served  weapons.  If  the  design  of  the  experiment 
is  sufficient  to  demonstrate  a  difference  between  these  two  interfaces,  then  we 
will  add  an  optical  head-tracking  device  that  allows  participants  to  observe 
peripheral  views  outside  the  laptop  window  screen  by  simply  turning  their  heads. 

B.  METHOD 

1.  Participants 

NPS  students  and  faculty  participated  in  the  simulation  experiment.  There 
were  three  different  sessions.  In  the  first  session,  participants  were  16  first  year 
NPS  MOVES  students  enrolled  in  an  introduction  virtual  systems  course. 

The  second  session  consisted  of  the  previous  16  students  and  an 
additional  10  students  and  faculty  who  participate  in  the  MOVES  weekly  brown 
bag  seminar.  The  additional  participants  gave  us  a  greater  opportunity  to  test 
our  methods,  procedures,  and  logic  on  a  larger  sample  size.  The  original  16 
students  were  the  focus  of  the  2nd  session  since  they  had  participated  in  the  first 
session.  We  did  this  so  we  could  use  a  within  subjects  design  to  evaluate 
performance  with  two  different  VBS2™  configurations. 

The  16  primary  participants  were  officers  ranging  from  0-1  2nd 
Lieutenants  (or  Lieutenant  JG)  to  0-5  Lieutenant  Colonels.  The  average  age 
was  32.1  (4.4),  with  a  range  between  27  and  42.  None  were  female.  Seven  of 
the  16  (44  percent)  identified  themselves  as  “Gamers”.  This  44  percent  is  much 
higher  than  typically  found  in  the  U.S.  Army.  According  to  Atkinson-Bonasio 
(2008)  about  “50  percent  of  young  enlisted  Soldiers  call  themselves  “gamers”  or 
are  familiar  with  the  mechanics  of  game  play.  At  the  officer  level  it  is  around  33 
percent.”  This  difference  was  to  be  expected  because  the  participants  were 
students  at  the  MOVES  Institute  at  NPS  and  this  curriculum  is  likely  to  draw 
someone  who  plays  video  games  in  their  free  time. 
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2.  Measures  and  Equipment 

The  project  used  16  Dell  Precision  M6300  laptops  from  a  suite  of  the 
Marine  Corps  Deployable  Virtual  Training  Environment  (DVTE)  package. 
Peripheral  equipment,  including  mice,  cables,  and  switches,  came  from  the 
standard  DVTE  package.  We  networked  all  sixteen  computers  together  with 
different  environments  using  four  D-LINK  DGS-2208  port  switches.  This  allowed 
us  to  push  configuration  files  and  download  participant  data  results  from  one 
central  location.  Participants  operated  as  individuals  in  the  evaluation. 

The  hardware  and  software  was  configured  by  the  participants  when  they 
arrived  at  the  MOVES  classroom.  The  participants  unloaded  the  laptops  from 
the  DVTE  boxes  and  physically  networked  the  computers  together.  They  then 
received  a  brief  where  they  uploaded  the  required  software  and  configured  their 
systems.  This  included  but  was  not  limited  to  configuring  the  time,  graphics 
display,  VBS2™  profile,  scenario,  InterfaceLib,  batch  files,  DIS  deploy,  disEntity 
and  VBSEntity  config  files.  The  participants  then  installed  and  configured  the 
NaturalPoint  TrackIRS™  and  SixAxis™  PS3™  interpreter  software  and  drivers. 
The  instruction  for  this  procedure  are  listed  in  Appendix  E.  DLI  Study  Computer 
Setup. 

The  last  step  was  to  start  the  scoring,  survey,  and  VBS2™  software.  First, 
we  started  the  VBS2™  software,  verify  settings,  and  upload  the  experimental 
training  and  evaluation  scenario.  Next,  VBS2™  was  minimized  and  the  Java 
JAR  file  was  started  to  administer  the  survey  and  record  the  scoring  data.  This 
resulted  in  all  the  required  software  running  in  the  background  and  the  first 
survey  question  waiting  to  be  answered  by  the  participant. 

We  used  three  surveys  to  gather  demographic  data  and  preference  data. 
The  first  survey  was  a  pre-exercise  survey  designed  to  gather  demographic  data 
on  the  participants.  The  second  was  a  post-exercise  survey  completed  after  the 
first  run  to  gather  participant  preference  data  on  the  user  initial  input 
configuration.  The  third  survey  was  a  post-exercise  survey  completed  after  the 
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final  run.  It  asked  the  same  questions  as  the  first  post-exercise  survey  but  added 
some  additional  questions  focused  on  identifying  participant  preference  between 
the  two  configurations. 

3.  Procedure 

a.  First  Session 

The  participants  were  first  briefed  on  the  purpose  and  generic 
details  of  the  study.  It  was  our  intent  to  completely  inform  the  participants  of  all 
details  of  the  experiment.  There  was  nothing  to  be  gained  by  limiting  their 
knowledge  of  the  experiment.  The  participants  were  then  instructed  to  read  the 
institutional  review  board  (IRB)  documentation  and  sign  the  IRB  informed 
consent  (Appendix  A)  stating  they  were  willing  participants.  There  were  no  balks 
at  signing  the  IRB  statement. 

After  signing  the  IRB  statement,  the  participants  completed  a 
demographic  survey  administered  by  our  automated  survey  program.  A  paper 
version  of  the  demographic  survey  used  for  the  pilot  study  is  included  in 
Appendix  E. 

When  everyone  completed  the  automated  demographic  survey,  we 
began  the  training  session.  The  training  session  consisted  of  an  information  brief 
followed  by  a  hands  on  exercise  on  how  to  use  the  standard  keyboard  and 
mouse  to  move  and  shoot  in  the  VBS2™.  Once  each  participant  had 
successfully  completed  each  training  task,  they  were  shown  a  video  of  a 
successful  run  on  the  test  course.  Again,  nothing  was  to  be  gained  by  keeping 
any  aspect  of  the  test  course  a  secret.  This  was  an  open  book  test  where  the 
limiting  factor  was  the  user-input  device  not  knowledge  of  the  test.  This  included 
a  detailed  discussion  on  exactly  how  their  performance  would  be  measured. 
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Figure  10.  Classroom  setup  for  keyboard  and  mouse  session 


The  participants  were  then  instructed  to  complete  the  test  course 
as  quickly  and  accurately  as  possible.  After  the  participants  completed  the  test 
course,  an  automated  trigger  closed  down  VBS2™  and  opened  a  post  exercise 
survey.  When  they  completed  the  post  exercise  survey  they  were  released  for 
the  day. 

Later,  we  downloaded  all  demographic  and  post  survey  text  files, 
the  automated  scoring  text  file,  and  the  after  action  review  video  file.  We  also 
reconfigured  the  workstations  to  support  using  the  PS3  game  controller  as  the 
primary  user-input  device. 

b.  Second  Session 

Two  weeks  later,  the  participants  returned,  received  an  abbreviated 
brief,  and  reviewed  and  signed  the  IRB  informed  consent  paperwork.  They 
received  a  second  training  session  using  the  game  controller  as  the  primary 
user-input  device.  Then  the  participants  were  tested  on  the  same  test  course 
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using  the  game  controller  instead  of  the  standard  keyboard  and  mouse 
configuration.  Upon  completion  of  the  test  course,  an  automated  trigger  closed 
the  VBS2™  application  and  opened  a  slightly  different  post-exercise  survey.  It 
asked  the  same  questions  as  the  first  post-exercise  survey  but  added  some 
additional  questions  focused  on  identifying  participant  preference  between  the 
two  configurations. 


Figure  1 1 .  Classroom  setup  for  PS3  game  controller  session 


During  the  second  session,  the  participants  ran  out  of  time  to 
complete  the  experiment.  This  was  a  problem  but  not  catastrophic.  We  were 
able  to  retrieve  three  of  the  four  data  sets  required  to  conduct  analysis.  We  were 
also  able  to  retrieve  the  pre  and  post-surveys  and  the  after  action  review  video 
from  each  participant.  Unfortunately,  the  scoring  interface  text  file  that  records 
the  trigger  times,  target  times,  and  ammunition  expenditures  were  lost.  This 
resulted  in  a  third  session  where  the  original  16  MOVES  student  participants 
reran  the  evaluation  portion  of  the  experiment. 
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c.  Third  Session 

One  week  later,  the  original  16  MOVES  student  participants 
returned  to  rerun  the  evaluation  test  course.  When  they  arrived,  they  reviewed 
and  signed  the  IRB  informed  consent  paperwork.  They  received  no  briefings  or 
any  additional  training.  They  completed  an  abbreviated  pre-survey  for  tracking 
purposes  only  because  we  already  had  their  pre-survey  data  from  earlier 
experiments.  Then,  the  participants  were  tested  on  the  same  test  course  using 
the  game  controller.  Upon  completion  of  the  test  course,  a  automated  trigger 
closed  the  VBS2™  application  and  opened  the  slightly  different  post-exercise 
survey  described  above  in  'Second  Session.  Finally,  we  downloaded  all  survey 
and  scoring  data  and  returned  the  VBS2™  and  the  DVTE  laptops  to  their 
baseline  configurations. 

C.  RESULTS 

1.  Evaluate  the  Experimental  Data 

We  used  a  significance  level  of  0.05  for  all  analysis.  The  first  statistical 
analysis  we  conducted  was  a  paired  f-test  comparing  the  number  of  M2  and  M4 
rounds  expended  during  the  evaluated  exercise. 

Ho:  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  game  controller  did  not  change  the  mean  number  of  rounds  expended  (pd  =  0). 

Ha:  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  game  controller  did  change  the  mean  number  of  rounds  expended  (pd  ^  0). 


Sample 


Weapon 

Size 

KB  Mean  (SD) 

GC  Mean  (SD) 

t-test 

p-value 

M4  Rds 

13 

54.1  (17.5) 

41.9  (27.1) 

1.24 

0.23 

M2  Rds 

13 

434.7  (212.6) 

557.1  (292.7) 

-1.24 

0.23 

Table  4.  Mean  with  standard  deviation  (SD)  of  rounds  needed  to  complete 
weapons  engagements  with  keyboard  (KB)  and  game  controller  (GC) 
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The  p-values  are  large,  so  we  retain  the  null  hypothesis  and  conclude  that 
there  is  no  difference  between  configurations  when  using  the  number  of  rounds 
fired  as  a  measure  of  effectiveness  for  both  M4  and  M2. 

The  second  statistical  analysis  we  conducted  was  a  paired  f-test 
comparing  the  amount  of  time  required  to  effectively  engage  required  targets  in 
the  shoot  /  no  shoot  lane  with  the  M4  rifle  and  the  Battle  Positions  with  the  M2 
heavy  machine  gun. 

Ho:  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  game  controller  did  not  change  the  time  required  to  effectively  engage 
required  targets  (pd  =  0). 

Hg:  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  game  controller  did  change  time  required  to  effectively  engage  required 
targets  (pd  5^  0). 


Sample 


Weapon 

Size 

KB  Mean  (SD) 

GC  Mean  (SD) 

t-test 

p-value 

M4  Range 

13 

63.9  (13.6) 

34.3  (7.3) 

8.55 

<0.001 

M2  Range 

13 

169.5(13.0) 

102.2  (55.9) 

3.32 

0.006 

Table  5.  Time  required  to  complete  weapons  engagements  (Time) 

The  P-value  is  well  below  our  a  of  0.05,  so  we  reject  the  null  hypothesis 
and  conclude  that  the  change  in  user-input  configurations  did  lead  to  a  change  in 
time  required  to  effectively  engage  required  targets. 

These  first  two  tests  had  no  credibility.  How  could  there  be  such  a  big 
difference  between  these  two  measures  of  effectiveness,  which  are  essentially 
measuring  the  same  thing?  When  we  dug  a  little  deeper,  we  found  that  our 
automated  scoring  application  was  correctly  capturing  the  time  data  but 
incorrectly  capturing  the  number  of  rounds  fired.  It  was  an  error  in  our  automated 
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scoring  application  source  code.  The  details  of  the  coding  error  and  our  solution 
is  fully  discussed  in  the  next  section  “Evaluating  the  study's  logic  and 
effectiveness.” 

The  third  statistical  test  we  conducted  was  a  paired  f-test  comparing  the 
amount  of  time  required  to  complete  the  mounted  and  dismounted  portions  of  the 
test  course. 

Hq:  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  PS3  game  controller  did  not  change  the  time  required  to  complete  the  test 
course  (pd  =  0). 

Ha!  The  change  in  the  user-input  configuration  from  keyboard  and  mouse 
to  PS3  game  controller  did  change  time  required  to  complete  the  test  course  (pd 
^0). 


Sample 


Weapon 

Size 

KB  Mean  (SD) 

GC  Mean  (SD) 

t-test 

p-value 

Mounted 

13 

103.8  (17.9) 

49.1  (8.7) 

10.15 

<0.001 

Dismounted 

13 

133.6  (25.8) 

83.81  (29.1) 

4.25 

0.001 

Table  6.  Time  required  to  complete  the  test  course  (Time) 

The  P-value  is  small,  so  we  reject  the  null  hypothesis  and  conclude  that 
the  change  in  user-input  configurations  did  lead  to  a  change  in  time  required  to 
complete  the  test  course  for  mounted  and  dismounted. 

2.  Evaluate  the  Study's  Logic  and  Effectiveness 
a.  Surveys 

Previous  NPS  VBS2™  thesis  experiments  used  paper  surveys  that 
allowed  participants  to  simply  not  complete  the  questions  resulting  in  large  gaps 
in  data  for  analysis.  As  a  result,  it  was  highly  recommended  that  we  use  an 
automated  survey  that  required  participants  to  enter  meaningful  answers,  even  if 

the  entry  is  “do  not  care  to  respond”.  When  we  coded  the  Java  JAR  file,  we 
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added  error  checking  within  the  methods  asking  the  questions  to  prevent 
erroneous  inputs.  The  JAR  file  created  to  gather  survey  information  was 
successful  but  far  from  perfect.  In  the  first  session  we  identified  that  some  of  the 
error  checking  did  not  allow  for  legitimate  answers  in  the  pre  survey  used  to 
collect  demographic  data.  It  was  a  simple  fix  to  adjust  the  values  used  in  error 
checking. 

The  second  issue  was  that  participants  who  did  not  want  to  answer 
a  question  could  close  the  dialog  box  and  continue  with  the  survey  as  stated  in 
the  IRB  approved  consent  form.  This  completely  defeats  the  purpose  of  the 
automated  survey.  The  issue  arose  when  the  participants  closed  the  dialog  box 
the  method  threw  an  exception  error  and  bypassed  the  error  checking.  The 
solution  was  to  add  an  error  checking  statement  outside  the  method  that  sends 
the  participant  into  a  continuous  loop  until  appropriate  data  is  entered. 

The  final  lesson  learned  in  regards  to  the  survey  was  a  reoccurring 
theme  in  the  pilot  study  and  probably  the  most  important  lesson  learned.  We 
overestimated  the  time  required  for  participants  to  complete  required  tasks.  It 
always  took  longer  to  complete  any  task  than  expected.  There  were  three 
reasons  for  this  planning  error.  The  first  reason  was  that  we,  as  the  researchers, 
were  too  close  to  the  problem  to  accurately  estimate  the  capabilities  of  a 
participant  who  had  never  seen  the  problem.  Secondly,  we  used  the  expected 
average  time  a  participant  would  require  to  complete  the  task  as  a  planning 
factor.  In  reality,  we  should  have  used  the  expected  time  for  the  slowest 
participant  to  complete  a  task  as  a  planning  factor.  The  final  reason  was  that  we 
were  not  exactly  sure  how  best  to  conduct  the  experiment,  so  we  purposefully 
errored  on  the  side  of  more  detail  instead  of  less  detail.  It  is  always  easier  and 
safer  to  take  out  aspects  of  the  experiment  after  the  pilot  study  than  to  add 
additional  requirements.  The  pre  and  post-surveys  were  both  longer  than 
necessary  and  were  streamlined  for  the  main  study. 
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b.  Project,  VBS2  ™,  Training,  and  Evaiuation  Course  Briefs 

The  next  set  of  lessons  involved  the  briefing  the  participants 
received  from  the  researchers.  The  briefing  was  too  long  and  did  not  include 
some  important  pieces  of  information.  These  lessons  were  identified  by  the 
participants  after  the  exercise  during  a  question  and  answer  period.  The  brief 
originally  provided  an  overview  of  commands  used  to  control  the  participant's 
avatar  with  the  keyboard  or  PS3  game  controller.  Afterward,  the  participant 
conducted  a  hands-on  training  exercise.  This  hands-on  training  exercise  was 
determined  to  be  redundant  and  confusing.  We  consolidated  the  overview  brief 
and  hands-on  training  exercise  resulting  in  a  more  efficient  and  effective  training 
session.  The  same  problem  was  identified  when  preparing  the  participants  to 
conduct  the  evaluation  portion  of  the  experiment.  The  participants  were  provided 
an  overview  brief  describing  the  evaluated  events  and  how  they  would  be  scored. 
Then  they  were  shown  a  video  of  what  the  correct  performance  looks  like. 
Again,  the  participants  identified  this  as  redundant  and  it  was  consolidated. 

After  we  completed  each  pilot  study  session,  we  conducted  an  after 
action  review  with  the  participants  that  provided  valuable  insight  into  what  was 
missing  from  the  briefing.  The  participants  recommended  that  we  include  a 
picture  of  each  target  during  the  overview  briefs.  There  initially  was  some 
confusion  on  the  difference  between  enemy  and  friendly  targets  during  the 
shoot/no  shoot  lane.  The  blue  targets  were  designated  as  friendly  and  the  green 
targets  with  yellow  Xs  were  designated  as  enemy.  There  was  also  confusion  on 
what  constituted  a  hit  when  engaging  the  insurgent  targets.  Sometimes  the 
insurgents  were  wounded,  but  did  not  die.  The  wounded  insurgents  are 
registered  as  target  hits  in  the  automated  scoring  application.  Participants 
repeatedly  asked  about  vehicle  rear  view  mirrors  and  if  the  user's  vehicle  can 
become  inoperable  due  to  sustaining  damage.  There  are  no  rear  view  mirrors 
and  the  vehicles  can  become  inoperable  during  the  conduct  of  the  training  and 
evaluation  exercise.  The  participants  recommended  that  we  clarify  these  vehicle 
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limitations  in  the  overview  brief.  We  also  added  a  hidden  spare  vehicle  in  the 
training  scenario  in  case  a  participant's  vehicle  became  inoperable. 

c.  Keyboard  and  Game  Controller  Function  Mapping 

We  also  learned  a  lot  about  the  mapping  of  the  functions  to  the 
keyboard  and  game  controller  controls.  Our  intent  was  to  limit  the  commands  to 
only  what  was  necessary  to  complete  the  evaluation  portion  of  the  experiment. 
In  the  first  session,  we  included  the  'toggle  weapons'  command.  The  'toggle 
weapons'  command  enables  participants  to  change  their  weapons  from  semi¬ 
automatic,  to  burst,  or  use  grenades.  This  command  confused  the  participants 
and  provided  no  value  to  the  exercise  as  we  want  the  user  to  only  use  semi¬ 
automatic.  We  removed  the  'toggle  weapons'  command  to  prevent  the 
confusion. 

The  second  issue  was  that  there  is  a  'free  look'  option  that  was  not 
made  available  to  participants.  In  planning,  we  did  not  think  it  was  necessary. 
The  'free  look'  functionality  enables  the  player  to  look  freely  around  the 
environment  without  changing  the  orientation  of  the  body.  It  allows  the  player  to 
move  their  head  but  not  their  body.  This  is  particularly  useful  when  driving  a 
vehicle  because  it  allows  the  player  to  look  outside  and  through  back  windows. 
The  TrackIR:  5  Head  Tracker  maps  to  this  specific  functionality.  After  reviewing 
the  performance  of  the  participants  on  the  combat  parking  event,  we  saw  that 
this  command  would  provide  value  to  participants  during  the  combat  parking 
event.  Furthermore,  it  would  be  disingenuous  to  introduce  the  TrackIR;  5  Head 
Tracker  capability  and  not  allow  the  same  capability  within  the  keyboard  and 
game  controller  configurations.  We  added  the  'free  look'  functionality  to  the  PS3 
game  controller  by  mapping  it  to  the  Circle  button.  This  was  a  better  solution  but 
not  quite  right.  During  the  PS3  game  controller  session,  participants  would 
unknowingly  hit  the  circle  button  and  activate  the  'free  look'  functionality.  This 
completely  disoriented  the  participant.  The  circle  button  is  on  the  far  right  side  of 
the  controller  and  located  directly  under  the  thumb.  This  location  often  resulted 
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in  the  accidental  activation  of  the  circle  button.  This  problem  is  further 
exacerbated  because  the  participant's  screen  does  not  change  when  the  button 
is  pressed  and  goes  unnoticed  until  the  participant  attempts  to  fire  their  weapons. 
We  moved  the  mapping  of  the  'free  look'  to  the  square  button.  This  button  is 
towards  center  of  the  game  controller  and  is  less  likely  to  get  accidently  pressed. 

The  third  issue  was  raised  by  participants  who  spend  an  exorbitant 
amount  of  time  playing  video  games,  and  have  very  specific  preferences.  They 
asked  about  the  ability  to  change  the  PS3  game  controller  to  “inverted.”  We  did 
not  know  about  inverted  controls  so  we  did  not  implement  this  functionality. 
Inverted  controls  is  a  setting  that  enables  you  to  press  down  and  look  up  or  vice 
versa.  According  to  our  participants  and  Andy  Robertson  of  GamePeople,  once 
a  gamer  discovers  inverted  controls  “there  is  no  going  back  for  these  gamers,  it 
becomes  so  engrained  in  their  playing  psyche  that  any  game  without  it  became 
almost  unplayable”  (Robertson  2006).  Our  participants  agreed.  One  of  our 
participants  compared  it  to  being  left  or  right  handed.  He  also  added  that  an 
experienced  gamer  might  actually  perform  worse  than  a  novice  if  not  provided 
with  an  inverse  controls  option.The  following  is  an  excerpt  from  an  Andy 
Robertson  2006  article  explaining  why  some  gamers  have  such  strong  opinions 
on  inverted  controls. 

“The  question  is  where  does  the  player  put  their  consciousness  in 
relation  to  the  controller.  What  part  of  their  body  is  the  joypad  controlling,  their 
arm,  their  head;  and  how  is  this  control  translated  to  that  body  part”  (Robertson 
2006). 

Behind:  If  the  player  feels  they  are  controlling  movement  from 
behind  their  head  with  joypad  ,  they  are  likely  to  find  an  inverted 
j  control  scheme  works  best  for  them.  Pulling  back  on  their  stick 
therefore  tilts  their  head  up  and  should  move  the  play  field  up. 

Figure  12.  Behind  (From  Robertson,  2006). 
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In  front:  If  the  player  feels  they  are  controlling  movement  in  front 
of  their  head,  they  are  likely  to  find  a  non-inverted  control  scheme 
works  best  for  them.  Pulling  back  on  their  stick  therefore  pulls 
their  head  down  from  the  front  and  should  move  the  play  field 
down. 


Figure  13.  In  front  (From  Robertson,  2006). 


Robertson  hypothesizes  that  extroverted  players  prefer  the  feeling 
of  controlling  the  gamer  in  front  of  themselves.  Whereas  an  introverted  player 
would  prefer  to  control  the  game  from  a  safe  location  behind  themselves.  There 
seems  to  be  no  right  answer  but  only  an  issue  of  individual  preference.  Some  of 
our  participants  believed  that  pilots  and  gamers  who  play  flight  simulations 
games  prefer  inverted  controls  because  it  is  more  representative  to  aircraft 
controls.  Regardless  of  the  reason,  we  introduced  inverted  controls  in  the  main 
study  as  an  option  as  not  to  hinder  gamers  performance  in  this  segment. 

d.  After  Action  Review  Recording 

The  first  session  of  the  pilot  study  also  reinforced  a  lesson  we  had 
learned  from  earlier  experiments.  The  more  you  can  automate  the  better.  Every 
time  a  researcher  depends  on  the  participant  to  manually  perform  a  task 
essential  to  the  experiment,  there  is  a  huge  risk  for  problems.  When  we 
designed  the  experiment,  we  automated  the  pre  and  post  surveys,  the  scoring 
application,  and  the  start  of  the  after  action  review  video  recording.  We  were  not 
able  to  automate  the  saving  of  the  after  action  review  video  recording  and 
depended  on  the  participants  to  save  this  file  at  the  end  of  the  session.  As  a 
result,  we  lost  nearly  20  percent  of  the  recorded  after  action  review  videos.  We 
contacted  the  VBS2™  technical  assistance  and  after  numerous  emails,  we 
discovered  a  procedure  that  automated  the  saving  of  the  after  action  review 
video  file.  The  procedure  involved  linking  a  VBS2™  trigger  to  a  script  that  saved 
the  after  action  review  without  placing  any  requirement  on  the  participant.  This 
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should  have  been  an  easy  fix  but  their  recommended  solution  did  not  work  on 
our  system  and  instead  presented  another  learning  opportunity. 

e  VBS2™  Version  Control 

The  next  learning  “opportunity”  concerns  VBS2^''^  software  version 
control.  There  are  at  least  3  versions  of  the  VBS2™  software  in  the  military 
today.  VBS2™  (version  2.21)  was  originally  delivered  to  the  U.S.  Marine  Corps 
as  part  of  the  Deployable  Virtual  Training  Environment  (DVTE)  Suite.  This 
version  has  a  Hardware-Assisted  Software  Protection  (HASP)  Key  issue  that 
results  in  VBS2^''^  randomly  failing.  We  have  had  five  of  the  32  DVTE  laptops  fail 
due  to  this  HASP  Key  error.  The  U.S.  Marine  Corps  and  Bohemia  Interactive  are 
well  aware  of  the  problem  and  provided  us  a  solution.  Their  recommended 
solution  was  to  reinstall  the  entire  hard  drive  with  a  clean  image  of  the  DVTE 
software  suite.  This  solution  was  not  optimal  but  we  had  no  choice.  We  also  had 
no  removable  hard  drive  with  a  clean  DVTE  master  upload  image  to  fix  the 
laptops  that  failed  due  to  the  HASP  Key  error.  We  obtained  a  clean  DVTE 
master  upload  image  on  a  removable  hard  drive  from  the  USMC  Program 
Manger  of  Training  Systems. 

After  receiving  the  removable  hard  drive  with  the  clean  DVTE 
image  we  reinstalled  the  DVTE  software  upload  onto  the  failing  laptops.  This 
fixed  the  HASP  Key  error.  The  new  DVTE  software  suite  was  delivered  with 
VBS2™  (version  1.3).  This  version  has  all  the  basic  startup  modes  such  as 
VBS2™  User,  VBS2™  Administrator,  and  VBS2™  Windowed.  But,  it  did  not  have 
VBS2™  LVC  Game.  We  needed  VBS2™  LVC  Game.  The  VBS2™  LVC  Game 
includes  the  Calytrix  add  on  that  provides  a  bi-direction  DIS/HLA  gateway  for 
connecting  VBS2™  into  existing  defense  simulations  systems  such  as  Joint 
Conflict  and  Tactical  Simulation  (JCATS),  Joint  Semi-Automated  Forces  (JSAF), 
or  One  Semi-Automated  Forces  (OneSAF)  (see  Figure  14). 
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Figure  14.  VBS2™  Network  Diagram.  (From  VBS2™  Users  Manual,  2010) 


We  needed  the  VBS2TM  LVC  Game  because  it  is  the  only  start  up 
mode  that  produces  and  sends  out  the  DIS  packets  to  a  network.  Our  automated 
scoring  applications  captures  these  DIS  packets  and  then  outputs  the  data  to  a 
text  file  that  is  later  used  to  score  the  participant's  evaluation  run.  Without  this 
capability,  we  would  need  to  manually  score  each  individual  run  by  observing  the 
recorded  after  action  review.  This  was  a  viable  backup  but  extremely  time 
consuming.  It  would  introduce  human  error  into  our  data  that  the  automated 
scoring  system  would  prevent.  We  needed  to  find  a  better  way. 

In  early  2009,  the  U.S.  Army  also  purchased  the  rights  to  VBS2TM 
and  is  currently  fielding  it  to  the  force.  We  requested  and  received  VBS2TM 
(version  1.23)  from  the  U.S.  Army's  National  Simulation  Center  (NSC)  in  late 
2009.  We  installed  it  on  one  DVTE  system  and  found  it  essentially  the  same  as 
the  U.S.  Marine  Corps  version  1.21  but  with  a  more  robust  PBO  library.  The 
PBO  library  contains  the  3  dimensional  characters,  vehicles,  weapons,  and 
building  with  their  associated  database  of  performance  parameters.  We  originally 
decided  not  to  upgrade  the  U.S.  Marine  Corps'  VBS2TM  (version  1.21)  with  the 
U.S.  Army's  VBS2TM  (version  1.23)  because  we  did  not  believe  the  value  of  the 
additional  PBO  library  outweighed  the  risk  associated  with  uploading  a  new 

version  of  VBS2TM.  After  all,  the  U.S.  Marine  Corps's  VBS2TM  (version  2.21) 
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was  working  well.  This  all  changed  when  we  received  the  clean  DVTE  master 
upload  containing  the  new  VBS2TM  (version  1.3)  that  did  not  possess  the 
VBS2TM  LVC  Game.  We  decided  to  assume  the  risk  and  upload  the  DVTE 
systems  with  the  U.S.  Army's  VBS2TM  (version  2.23)  with  LVC  Game. 

The  VBS2™:  Army  (version  2.23)  fixed  two  persisting  problems. 
First,  it  had  the  VBS2™  LVC  Game  to  run  our  scoring  application.  Secondly,  the 
VBS2:Army™  (version  2.23)  also  fixed  the  challenge  of  automating  the  saving 
procedure  of  the  recorded  after  action  review  files.  The  reason  the  Bohemia 
Interactive  technical  support  solution  did  not  work  on  our  system  was  because 
we  were  using  two  different  version  of  VBS2™.  Once  we  upgraded  to  the  newer 
version,  their  recommended  solution  worked  perfectly. 

One  of  the  drawbacks  of  upgrading  versions  was  the  compatibility 
of  our  VBS2™  training  scenario  file.  The  training  scenario  file  we  built  using  the 
original  VBS2™  (version  1.21),  did  not  transfer  perfectly  to  the  upgraded  VBS2™ 
(version  1.23).  This  did  not  require  a  complete  rebuild  but  did  require  significant 
trial  and  error  to  find  the  differences.  A  few  examples  of  the  differences  were 
links  between  some  actions  and  triggers  that  were  broken,  some  objects  were 
not  “snapped  to  surface”  (flush  with  the  ground),  and  a  few  objects  were 
repositioned  or  looked  different.  None  of  these  were  difficult  to  fix,  but  all  were  a 
challenge  to  find. 

f.  Automated  Scoring  Application 

The  application  created  to  gather  data  during  the  exercise  worked 
well  but  was  not  perfect.  The  scoring  application  accurately  captured  all  the 
targets  and  trigger  times  required  to  conduct  our  analysis  but  it  did  not  accurately 
capture  the  number  of  M4  rifle  rounds  fired  or  M2  machinegun  rounds  fired.  We 
examined  the  code  and  discovered  we  were  counting  the  number  of  rounds  fired 
in  both  the  training  and  evaluation  phases  of  the  experiment.  We  removed  the 
code  that  counted  rounds  expended  from  the  training  phase.  This  fixed  the  M4 
rifle  counting  problem  but  not  the  M2  machinegun  round  count  problem. 
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The  M2  problem  was  caused  by  how  DIS  packets  are  sent  over  the 
network.  M4  rifle  DIS  packets  are  coded  as  “fire”  PDU  whereas  the  M2  dis 
packets  are  coded  as  “detonation”  PDU.  We  were  able  to  capture  and  count  the 
M4  “fire”  PDU  but  we  could  not  do  the  same  for  M2  “detonation”  PDUs  because 
there  were  too  many  events  that  produced  “detonation”  that  we  could  not  sort 
through  all  the  “detonations”  and  accurately  extract  only  the  detonations 
produced  by  firing  with  the  M2.  After  considerable  discussion,  we  decided  that 
this  was  not  actually  a  problem.  The  M4  rifle  is  a  precision  weapon  and  counting 
rounds  is  important,  whereas  the  M2  heavy  machine  gun  is  an  area  fire  weapon 
where  time,  not  rounds  expended,  is  the  accepted  measure  of  effectiveness. 

g.  The  VBS2™  Training  and  Evaiuation  Courses 

There  were  more  lessons  learned  but  the  last  set  of  lessons  we  will 
discuss  here  was  with  the  VBS2™  training  scenario  file  itself.  This  is  the  actual 
virtual  environment  we  created  using  the  VBS2™  mission  editor  to  train  and 
evaluate  our  participants.  Most  of  these  issues  were  found  during  the  analysis  of 
the  data.  First,  the  most  challenging  portion  of  the  course  was  the  combat 
parking  lane.  It  was  first  deemed  too  difficult,  and  then  we  realized  that  we 
trained  every  task  except  combat  parking.  The  solution  was  to  add  a  combat 
parking  lane  at  the  end  of  the  training  session  in  order  to  allow  participants  to 
practice  this  task  at  least  once  before  proceeding  to  the  evaluated  exercise. 

The  second  lesson  was  in  the  shoot  /  no  shoot  lane  in  which 
participants  were  to  walk  down  a  city  street  and  engage  four  enemy  targets,  then 
make  a  right  hand  turn,  and  engage  another  two  enemy  targets.  Nearly  40 
percent  of  the  participants  made  the  right  hand  turn  when  they  saw  the  ramp  lane 
with  green  arrows  and  bypassed  the  two  last  targets.  Because  of  this  problem 
and  for  the  sake  of  saving  time,  we  removed  the  last  two  shoot  /  no  shoot 
engagements.  We  also  received  feedback  from  participants  stating  that  the  task 
was  too  repetitive;  they  indicated  they  had  achieved  their  purpose  with  the  first 
four  engagements,  and  thus,  the  last  two  were  unnecessary. 
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The  other  problem  with  the  shoot/no  shoot  lane  was  that  we  had  no 
time  trigger  between  the  combat  parking  lane  and  the  shoot  /  no  shoot  lane. 
Therefore,  we  could  not  establish  a  start  time  for  the  dismounted  portion  of  the 
course.  We  adjusted  one  of  the  time  triggers  to  fix  this  problem. 

In  the  first  session,  two  participants  got  lost  at  the  transition  point. 
As  a  result,  the  obstacle  course  was  painstakingly  designed  to  prevent 
participants  from  getting  lost.  The  participants  were  essentially  boxed  into  the 
scenario,  and  not  given  the  opportunity  to  leave  the  desired  route.  The  one 
exception  was  at  the  transition  point  where  the  participant  exits  the  vehicle  and 
begins  the  dismounted  portion  of  the  exercise.  The  two  participants  who  got  lost 
took  the  same  wrong  road  at  the  transition  point.  This  was  fixed  by  emplacing  a 
series  of  15-foot  T-wall  barricades  to  block  participants  from  getting  off  the 
desired  route. 

The  last  problem  was  with  the  final  three  targets  on  the  M4  rifle 
range.  The  distance  to  the  targets  were  too  great  to  effectively  engage  with  the 
PS3  game  controller  and  they  were  very  difficult  to  identify.  We  fixed  this  by 
moving  the  targets  in  about  40  meters  and  placing  yellow  range  fans  on  the  left 
and  right  limits  of  the  range.  We  also  created  range  fans  to  the  training  M4  rifle 
range  and  moved  some  of  the  practice  targets  further  away  to  provide  a  better 
training  opportunity  for  the  participants. 

D.  DISCUSSION 

The  results  of  the  study  demonstrated  that  several  logistical  issues  had  to 
be  resolved,  but  that  our  logic  was  sound  and  the  PS3  game  controller  was  a 
superior  user-input  device.  One  factor  that  we  did  not  control  for  was  sequence. 
All  of  our  participants  first  used  the  keyboard  and  mouse  configuration  followed 
by  the  game  controller  configuration.  What  we  did  not  uncover,  was  how  much 
the  sequence  skewed  our  data;  but  we  suspect  it  had  a  significant  effect.  This 
resulted  in  a  change  in  the  design  of  our  experiment  to  incorporate  the  sequence 
as  an  independent  variable. 
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We  believe  that  this  pilot  study  was  successful  by  gathering  sufficient 
information  to  test  the  logic  of  our  study.  It  also  provided  insight  to  where  our 
logic  and  execution  was  flawed.  Finally,  it  provided  us  with  an  abundance  of 
lessons  learned  that,  when  incorporated  into  our  larger  thesis  study,  will  improve 
the  quality  and  efficiency  of  the  research. 
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V.  DEFENSE  LANGUAGE  INSTITUTE  USER-INPUT  DEVICE 

EXPERIMENT 


A.  BACKGROUND 

1.  Introduction 

The  purpose  of  the  Defense  Language  Institute  (DLI)  experiment  was  to 
evaluate  the  performance  and  preferences  of  Soldiers  using  different  user-input 
devices.  All  of  the  lessons  learned  during  the  pilot  study,  described  in  Chapter  IV 
were  corrected  for  this  experiment.  Through  coordination  with  the  229th  Military 
Intelligence  Battalion  Commander  and  his  operations  officer,  we  were  able  to 
conduct  our  experiment  on  a  sample  from  the  population  of  interest:  young  active 
duty  Soldiers. 

The  study  required  that  two  experiments  be  run  simultaneously.  The  only 
difference  between  the  two  experiments  was  how  the  user-input  device 
configuration  varied.  The  first  experiment  was  designed  to  test  the  hypothesis: 
does  adding  a  commercial  head-tracking  device  to  virtual  desktop  simulations 
improve  military  training  effectiveness  by  providing  a  more  intuitive  user-input 
interface.  The  second  experiment  was  designed  to  test  the  hypothesis:  does 
adding  a  commercial  game  controller  device  to  virtual  desktop  simulations 
improve  military  training  effectiveness  by  providing  a  more  intuitive  user-input 
interface.  In  study  one  of  our  experimental  design,  we  keep  the  keyboard  or  PS3 
game  controller  consistent  but  change  the  head-tracking  device.  In  the  second 
study  we  keep  the  head  tracker  consistent  and  change  the  keyboard  or  game 
controller.  This  experimental  design  also  controls  for  sequence  of  configuration 
seen  by  the  participants  (Table  7). 
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Group 

Input 

Device 

Head 

T  racker 

Input 

Device 

Head 

T  racker 

study  1 

(Head  Tracker) 

1 
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— 

- 
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2 

KB 

+ 

- 

KB 

— 

3 

PS3 

— 

- 

PS3 

+ 

4 

PS3 

+ 

- 

PS3 

— 

I^H 

■ 

study  2 

(Input  Device) 

5 

PS3 

+ 

KB 

+ 

6 

KB 

+ 

- 

PS3 

+ 

7 

PS3 

— 

- 

KB 

— 

8 

KB 

— 

- 

PS3 

— 

Table  7.  Experimental  Design 


As  an  exploratory  hypothesis,  we  examined  if  factors,  such  as  age  and 
map  representation,  influence  the  effect  of  user  input  device  on  performance.  It 
would  be  valuable  to  identify  what  type  of  Soldier  could  benefit  most  from 
different  user  input  configurations  or  map  representations.  We  also  thought  it 
would  be  useful  to  know  if  participants  could  predict  their  performance  based  on 
input  device  preference.  This  would  give  credence  to  the  thought  that  one 
configuration  is  not  superior  to  another,  but  superiority  is  based  on  the  individual 
and  not  the  configuration.  The  final  recommendation  might  be  to  give  the 
Soldiers  ability  to  chose  a  configuration,  for  they  will  know  what  is  best  for  them. 
The  analysis  was  only  conducted  in  the  game  controller  study.  Participants  could 
not  provide  a  head  tracker  preference  since  none  were  familiar  with  the  new 
technology.  Whereas,  everyone  was  familiar  with  a  game  controller  and  30  of 
the  33  participants  had  a  preference  between  the  game  controller  and  the 
keyboard  and  mouse  configuration. 
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2. 


Research  Questions 


1.  Do  user-input  devices  impact  military  training  performance  when 
used  with  a  desktop  virtual  training  simulation? 

2.  Do  soldiers  prefer  one  user-input  device  over  another  when  using  a 
desktop  virtual  training  simulation? 

3.  Hypotheses 

1.  The  addition  of  a  commercial  head-tracking  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training 
simulation  and  be  preferred  over  no  head-tracking  device. 

2.  The  addition  of  a  commercial  game  controller  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training 
simulation  and  be  preferred  over  the  standard  keyboard  and 
mouse. 

4.  Exploratory  Hypotheses 

Do  factors,  such  as  age  and  map  representation,  influence  the  effect  of 
user  input  device  on  performance  and  preference? 

B.  METHOD 

1.  Participants 

We  first  approached  the  U.S.  Army  Battalion  Commander  at  the  Defense 
Language  Institute  six  months  prior  to  the  experiment  to  formally  brief  and 
request  his  support.  We  explained  the  purpose  and  timeline  of  our  study, 
logistical  support  requirements,  requirements  for  his  Soldiers,  and  how  our  study 
would  benefit  the  U.S.  Army.  He  agreed  that  our  research  was  important  and  he 
would  support  our  efforts.  He  then  instructed  his  operations  officer  to  contact  his 
company  commanders  and  ask  for  volunteers. 
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The  participants  for  this  study  were  all  U.S.  Army  Soldiers  enrolled  in 
language  training  at  the  Defense  Language  Institute  (DLI).  They  were  all 
volunteers  and  were  not  provided  any  compensation.  The  Soldiers  did  have  a 
requirement  to  perform  a  certain  amount  of  community  service.  The  unit  chain  of 
command  determined  that  participation  in  our  study  warranted  community 
service  credit. 

We  had  a  total  of  53  participants.  All  the  participants  were  enlisted  U.S. 
Army  Soldiers  ranging  from  Privates  (E-1)  to  Staff  Sergeants  (E-6).  The  average 
age  was  24.5  (5.2),  with  a  range  between  18  and  39.  Six  of  the  53  participants 
were  female. 

Eighty-four  percent  (45  of  53)  of  the  participants  stated  they  had  some 
type  of  computer  gaming  experience.  Seventy-seven  percent  (41  of  53)  stated 
they  had  console  gaming  experience  with  an  average  of  9.7  (8.4)  years  of 
experience.  Seventy  percent  (37  of  53)  stated  they  had  PC  gaming  experience 
with  an  average  of  7.7  (8.4)  years  of  experience. 

Thirty-four  of  the  53  (64  percent)  identified  themselves  as  'gamers'. 
Based  on  discussion  with  pilot  study  participants  we  defined  a  “gamer”  as 
anyone  who  spent  more  than  5  hours  a  week  playing  video  games.  Our  pilot 
study  participants  believed  that  someone  who  played  video  games  more  than 
five  hours  a  week  would  have  a  marked  advantage  in  our  study. 

These  levels  of  gaming  experience  are  higher  than  you  would  typically  find 
in  the  U.S.  Army  according  to  past  studies.  Roger  Smith  is  the  Chief  Technology 
Officer  for  the  U.S.  Army  Program  Executive  Office  for  Simulation,  Training,  and 
Instrumentation  (PEO-STRI).  Dr.  Smith  estimates  about  “50  percent  of  young 
enlisted  Soldiers  call  themselves  'gamers'  or  are  familiar  with  the  mechanics  of 
game  play.  At  the  officer  level  it  is  around  33  percent”  (Atkinson-Bonasio,  2008). 
This  difference  was  to  be  expected  since  the  participants  were  self  selected 
volunteers  and  'gamers'  are  more  likely  to  volunteer  their  time  to  play  a  video 
game  for  training. 
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Dr.  Smith  made  this  estimate  2  years  ago  and  times  are  changing  quickly. 
As  time  passes,  it  is  not  unreasonable  to  assume  that  more  Soldiers  are  entering 
the  military  with  gaming  experiment  while  older  soldiers  without  gaming 
experience  retiring.  The  most  likely  of  reason  for  this  difference  is  our  pool  of 
participants  were  simply  different  then  the  U.S.  Army  as  a  whole.  Our  pool  of 
participants  consisted  of  linguist  Soldiers  attending  language  training  at  the 
Defense  Language  Institute  (DLI).  The  Armed  Services  Vocational  Attitude 
Battery  (ASVAB)  Skilled  Technical  (ST)  test  scores  required  to  enlist  as  a  linguist 
are  much  higher  than  the  majority  of  military  occupational  specialties.  Soldiers 
with  higher  ASVAB  ST  test  scores  are  more  likely  to  have  access  to  computer 
based  technologies. 

2.  Measures 

a.  Equipment 

In  the  weeks  prior  to  the  DLI  study  we  configured  all  the  systems  at 
NPS.  This  configuration  included  but  was  not  limited  to  configuring  the  graphics, 
profile,  scenario,  batch,  configuration,  and  Java  jar  files.  We  then  installed  and 
configured  the  NaturalPoint  TrackIRS™  and  SixAxis™  PS3^''^  interpreter  software 
and  drivers.  The  instruction  for  this  procedure  are  listed  in  Appendix  D.  DLI  Study 
Computer  Setup. 

Prior  to  the  arrival  of  the  participants,  we  prepared  the  229th 
Military  Intelligence  Battalion  conference  room  for  our  experiment.  This 
preparation  began  with  setting  up  and  networking  twenty  Dell  Precision  M6300 
laptops  from  a  suite  of  the  U.S.  Marine  Corps  DVTE  package.  The  computers 
were  networked  using  D-LINK  DGS-2208  port  switches.  Peripheral  equipment, 
including  mice,  cables,  and  switches,  came  from  the  standard  DVTE  package. 
The  user-input  devices  included  Sony  PS3™  game  controllers  and  NaturalPoint 
TrackIRS™  head  trackers. 
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All  the  computers  were  run  in  administrator  mode  with  three 
software  application  running.  The  three  software  application  were  VBS2™, 
NaturalPoint  TrackIR  5™,  and  the  SixAxis™  PS3™  Interpreter.  The  VBS2™ 
profile  files  were  updated  prior  to  each  experimental  group.  The  first  profile  file 
was  the  standard  file  but  slightly  modified  by  disabling  some  of  the  keyboard 
capabilities  to  ensure  that  the  keyboard  did  not  provide  an  advantage  over  the 
game  controller.  The  second  profile  file,  disabled  all  the  keyboard  functions  to 
ensure  that  the  participants  were  forced  to  use  only  the  game  controller  and  did 
not  short  cut  the  experiment  by  using  keyboard  commands.  Lastly,  we  began  the 
Java  jar  file  to  administer  the  survey  and  record  the  scoring  data. 

b.  Scenario 

The  evaluation  course  began  with  a  mounted  exercise  where  the 
participant  drove  a  HUMMV  and  fired  a  crew  served  .50  Cal  machine  gun.  This 
was  followed  by  a  dismounted  exercise  where  the  participant  was  required  to 
negotiate  several  events  and  ended  on  a  M4  rifle  range  (Figure  15).  The 
participant  started  at  the  designated  start  point  (1),  drove  through  a  serpentine 
course  (2),  and  occupied  the  first  vehicle  battle  position  (3).  There,  the 
participant  switched  from  the  driver’s  seat  to  the  gunner’s  station  and  engaged 
three  truck  targets.  Afterward,  the  participant  returned  to  the  driver’s  seat  and 
continued  on  the  driving  course.  The  participant  negotiated  a  90-degree  left  turn 
with  buildings  on  both  sides  of  the  road  (4).  Additionally,  two  civilian  vehicles 
approached  in  this  area,  and  the  participant  had  to  negotiate  around  the 
oncoming  traffic.  Once  through  the  built-up  area,  the  participant  occupies  a 
second  battle  position  (5),  and  engages  three  additional  truck  targets.  Once  the 
engagement  is  complete,  the  participant  continued  on  the  driving  course, 
negotiating  two  more  civilian  vehicles  worth  of  traffic.  At  the  end  of  the  mounted 
course  (6),  the  participant  combat  parks  the  HUMMV  by  backing  it  between  traffic 
barriers  and  cones.  Measures  of  performance  for  the  mounted  exercise  include 
the  following;  number  of  vehicle  collisions  with  scenery  objects  including  traffic. 
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number  of  times  all  four  wheels  of  the  vehicle  went  off  the  road,  number  of 
targets  hit,  number  of  rounds  expended,  and  time  for  each  portion  of  the  course 
(Table  8). 


1.  start  point 
2. Serpentine 

3.  Battle  Position  1 

4.  Urban  Driving 

5.  Battle  Position  2 

6.  Dismount  Point 

7.  Shoot /No  Shoot  Lane 

8.  Ramps  and  Mazes 

9.  Small  Arms  (M4)  Range 


9  -t 

.7*'' 

5 


Evaluation 

Course 


Figure  15.  Evaluation  Course  Layout. 


The  dismounted  portion  of  the  course  started  when  the  participant 
entered  the  'shoot  /  no  shoot'  event  (7).  This  event  was  in  a  urban  environment 
consisting  of  buildings  and  alleys  where  enemy  and  friendly  targets  were 
presented  to  the  participants.  The  alleys  were  on  opposite  sides  of  the  road, 
forcing  the  participant  to  look  fully  to  the  left  and  the  right  to  determine  which 
target  to  engage.  The  next  event  was  the  ramp  and  weave  course  (8)  where 
participants  walked  across  elevated  ramps  arranged  at  different  angles.  This 
was  followed  by  two  mazes.  One  maze  had  walls  high  enough  that  participants 
could  not  see  over  the  top  resulting  in  poor  awareness  of  their  surroundings  or 
direction.  The  other  maze  had  low  walls  so  that  participants  can  see  over  the  top 
of  the  whole  course.  At  the  end  of  the  second  maze,  the  participants  low  crawled 
through  a  tunnel,  stood,  and  ran  to  the  individual  shooting  range  (9).  There, 
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participants  engaged  four  near  (<50  meters)  and  three  distant  (>200  meters) 
personnel  targets.  Performance  measures  for  the  individual  portion  include: 
number  of  falls  off  the  ramps,  number  of  enemy  targets  hit,  number  of  friendly 
targets  hit,  number  of  rounds  expended,  and  time  for  each  portion  of  the  course. 
Table  8  summarizes  the  full  evaluation  course. 


Event 

Participant  Action 

Performance  Measure 

Mounted  weave 

Drive  around  barricades  as  quickly  as 
possible  without  wrecking  the  vehicle 

•  Time 

•  Numberof  vehicle  crashes 

Mounted  shooting 

Use  the  HMMWV  M2  to  destroy  3 

•  Time 

range,  Station  1 

enemy  trucks 

•  Number  targets  hit 

•  Number  rounds  expended 

Mounted  shooting 

Use  the  HMMWV  M2  to  destroy  3 

•  Time 

range,  Station  2 

enemy  trucks 

•  Number  targets  hit 

•  Number  rounds  expended 

Mounted  movement 

Move  from  mounted  movement 
course  start  to  stop,  with  time  at 
mounted  shooting  range  stations 
subtracted 

•  Time 

•  Number  scenery  objects  hit 

•  Number  times  vehicle  off  road 

Ability  to  observe 

Combat  park  vehicle 

Conduct  shoot  /  no  shoot  alleys 

•  Time 

•  Number  targets  hit 

•  Number  friendlies  hit 

Individual  movement 

Move  through  ramp  weave,  high 
weave,  and  low  weave 

•  Time 

•  Number  falls  off  ramp 

Individual  shooting 

Use  the  rifle  to  shoot  6  short  range 

•  Time 

range 

personnel  targets  and  3  long  range 
popup  targets 

•  Number  targets  hit 

•  Number  rounds  expended 

Table  8.  Interface  evaluation  course  station  guide 


c.  Surveys 

We  used  three  surveys  to  gather  demographic  and  preference 
data.  The  first  survey  was  a  pre-exercise  survey  designed  to  gather 
demographic  data  on  the  participants  (Appendix  G).  The  second  was  a  post¬ 
exercise  survey  completed  after  the  first  run  to  gather  initial  participant 
preference  data  on  the  first  configuration  (Appendix  H).  The  third  survey  was  a 
post-exercise  survey  completed  after  the  final  run.  It  asked  the  same  questions 
as  the  first  post-exercise  survey  but  added  some  additional  questions  focused  on 

identifying  participant  preference  between  the  two  configurations  (Appendix  I). 
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3. 


Procedure 


a.  Arrival 

When  the  participants  arrived  for  the  study  they  were  sequestered 
into  a  separate  room  until  all  had  arrived.  The  participants  were  asked  two 
questions  in  order  to  insure  the  experimental  groups  were  balanced.  The  first 
was  “do  you  consider  yourself  a  gamer?”  We  then  defined  a  “gamer”  as 
someone  who  plays  video  games  more  than  5  hours  per  week.  The  participants 
who  identified  themselves  a  gamers  were  then  evenly  distributed  over  each  of 
the  experimental  groups. 

The  second  question  asked  was  if  any  of  the  self  proclaimed 
“gamers”  preferred  inverted  controls  over  standard  controls.  We  felt  that  this  was 
an  important  distinction  that  we  learned  from  our  pilot  study  and  literature  review. 
Only  three  of  the  53  participants  stated  that  they  preferred  inverted  controls.  We 
assigned  these  participants  to  VBS2™  systems  configured  with  inverted  controls. 
A  third  split  was  made  on  sex  to  ensure  that  men  and  women  were  evenly 
distributed  over  each  control  groups.  We  only  had  six  women  participants,  so  six 
experimental  groups  had  one  female  and  two  groups  had  none. 

b.  Introduction  Brief,  Institutional  Review  Board  (IRB) 
Consent  Form,  and  Demographic  Survey 

The  introduction  brief  explained  purpose  and  generic  details  of  the 
study  (Figure  16).  It  was  our  intent  to  completely  inform  the  participants  of  all 
details  of  the  experiment.  There  was  nothing  to  be  gained  by  limiting  their 
knowledge  of  the  experiment.  The  participants  were  then  instructed  to  read  the 
internal  review  board  (IRB)  documentation  and  sign  stating  they  were  willing 
participants.  The  IRB  consent  form  can  be  found  in  Appendix  E.  There  were  no 
balks  at  signing  the  IRB  statement.  After  signing  the  IRB  statement,  the 
participants  completed  a  demographic  survey  administered  by  the  Java  file.  The 
demographic  survey  can  be  found  in  Appendix  E. 
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Figure  16.  DLI  Study  Introduction  Brief 


c.  Conduct  of  the  Exercise 

When  everyone  completed  the  demographic  survey,  we  began  the 
training  session.  The  participants  were  provided  a  map  and  imagery 
representation  of  the  training  and  test  course  (Appendix  F).  They  were  also 
provide  a  cheat  sheet  showing  the  keyboard  and  game  controller  commands  and 
shortcuts  (Appendix  G).  The  training  session  consisted  of  an  information  briefing 
followed  by  a  hands  on  exercise  on  how  to  move  and  shoot  in  the  VBS2™  virtual 
environment  as  an  individual  and  mounted  in  military  vehicle.  Once  each 
participant  had  successfully  completed  each  training  task,  they  were  shown  a 
video  of  a  successful  run  on  the  evaluation  course.  Again,  nothing  was  to  be 
gained  by  the  keeping  any  aspect  of  the  test  course  a  secret.  This  was  an  open 
book  test  where  the  limiting  factor  was  the  skill  of  the  participant,  not  knowledge 
of  the  test.  We  also  explained  how  their  performance  would  be  evaluated. 
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The  participants  were  then  instructed  to  complete  the  test  course 
as  quickly  and  accurately  as  possible  (Figure  17).  After  the  participants 
completed  the  test  course,  an  automated  trigger  closed  down  VBS2™  and 
opened  a  post  exercise  survey.  This  survey  was  designed  to  gain  immediate 
feedback  on  their  user-input  device  configuration.  When  they  completed  the  post 
exercise  survey  they  were  put  on  a  break.  The  breaks  varied  in  time  dependent 
on  how  quickly  they  finished  the  test  but  everyone  was  allowed  at  least  ten 
minutes.  During  the  break,  we  downloaded  all  demographic  and  post  survey 
files,  the  automated  scoring  files,  and  the  after  action  review  video  file.  We  also 
reconfigured  the  workstations  user-input  configurations  for  the  second  run. 


Figure  17.  DLI  Participants  Testing  on  the  PS3™  Game  Controller 

Configuration 

Each  participant  returned  to  the  same  workstation  and  receive  a 
second  training  session  on  a  different  user-input  configuration.  The  participant 
completed  the  same  test  course  using  a  different  user-input  configuration.  Upon 
completion  of  the  second  test  course,  a  trigger  close  the  VBS2™  application  and 
opened  a  second  post  exercise  survey.  This  survey  asked  the  same  questions 
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as  the  first  post-exercise  survey  but  added  some  additional  questions  focused  on 
identifying  participant  preference  between  the  two  configurations. 


C.  RESULTS 

1.  Head  Tracker  Study  Results 


Paired  t-test  were  used  to  test  hypotheses  regarding  overall  measure  of 
performance  and  individual  components  of  performance.  The  addition  of  the 
commercial  head-tracking  device  to  the  virtual  desktop  simulation  did  not 
improve  performance  in  a  virtual  training  simulation  and  was  not  preferred  over 
no  head-tracking  device.  The  participants'  overall  combined  scores  were 
significantly  worse  when  using  the  head  tracker  (Table  9).  The  lower  the  score 
the  better  the  performance. 


•  Ho:  The  addition  of  a  commercial  head-tracking  device  to  virtual 
desktop  simulations  will  not  improve  performance  in  a  virtual 
training  simulation  and  be  preferred  over  no  head-tracking  device. 

(Md  =  0) 

•  Ha!  The  addition  of  a  commercial  head-tracking  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training 
simulation  and  be  preferred  over  no  head-tracking  device,  (pd  >  0) 


Head  Tracker 
Overall  (Total) 


Sample  Mean  Score  Mean  Score 

Size  without  HT  (SD)  with  HT  (SD)  t-test  p-value 

22  17.43  (8.23)  22.18  (7.68)  -3.61  0.001 


Table  9.  Matched  Paired  T-test  Comparing  Overall  (Total)  Score  with  no  Head 
Tracker  (HT)  to  Scores  with  Head  Tracker  (Lower  is  better) 


The  participants  scores  did  not  improve  with  the  head  tracker  on  any 
event.  The  participants'  scores  on  four  of  the  six  events  were  significantly  worse 
when  using  the  head  tracker  and  the  other  two  events  were  inconclusive  (Table 
10).  The  configuration  without  the  head  tracker  demonstrated  improved 
performance  on  the  mounted  moving,  dismounted  moving,  dismounted  shooting 
and  the  shoot  house  events.  These  events  are  highlighted  in  yellow  bold  text  in 
Table  10.  The  mounted  shooting  and  parking  events  were  inconclusive.  The 
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parking  event  was  purposely  designed  to  take  advantage  specific  capabilities 
provided  by  of  the  head  tracker  and  it  also  did  not  produce  improved 
performance.  The  lower  the  score  the  better  the  performance. 


Sample  Mean  Score  Mean  Score 
Size  without  HT  with  HT 


Event 

(SD) 

(SD) 

t-test 

p-value 

Mounted  Moving 

22  2.84  (0.94) 

4.17  (2.27) 

-3.44 

0.001 

Mounted  Shooting 

22  3.25  (2.51) 

3.52  (2.23) 

-0.45 

0.328 

Dismounted  Moving 

22  2.27  (2.02) 

2.92  (1.97) 

-1.75 

0.047 

Dismounted  Shooting 

22  1.93  (0.84) 

2.84  (1.83) 

-2.51 

0.010 

Shoot  House 

22  2.43  (1.76) 

4.01  (1.91) 

-3.85 

0.001 

Parking 

22  4.69  (3.19) 

4.69  (3.51) 

0 

0.500 

Table  10.  Matched  Paired  T-test  Comparing  Scores  with  no  Head  Tracker  between 

Scores  with  Head  Tracker  (Lower  is  better) 


Head  Tracker  Comparision 


Mounted  Mounted  Dismounted  Dismounted  Shoot  House  Parking 

Moving  Shooting  MO'Ving  Shooting 

■  Mean  Score  without  Head  Tracker  ■  Moan  Score  with  Head  Tracker 


Figure  18.  Graph  Comparing  Mean  Event  Scores  with  no  Head  Tracker  to 

Scores  with  Head  Tracker  (Lower  is  Better) 


Finally,  in  the  post  survey,  1 1  of  the  22  (50  percent)  of  the  participants 

stated  they  preferred  the  addition  of  the  head  tracker.  The  other  1 1  participants 

stated  they  preferred  no  head-tracking  device. 
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2.  Game  Controller  Study  Results 

Paired  t-test  were  used  to  test  hypotheses  regarding  overall  measure  of 
performance  and  individual  components  of  performance.  The  addition  of  the 
commercial  game  controller  to  the  virtual  desktop  simulation  did  not  improve 
performance  in  a  virtual  training  simulation  and  was  not  preferred  over  the 
keyboard  and  mouse  configuration.  The  participants'  overall  combined  scores 
were  significantly  worse  when  using  the  game  controller  (Table  11).  The  lower 
the  score  demonstrates  the  better  the  performance. 

•  Ho:  The  addition  of  a  commercial  game  controller  device  to  virtual 
desktop  simulations  will  not  improve  performance  in  a  virtual 
training  simulation,  (pd  =  0) 

•  Hg:  The  addition  of  a  commercial  game  controller  device  to  virtual 
desktop  simulations  will  improve  performance  in  a  virtual  training 
simulation,  (pd  >  0) 


Game 

Controller 

Overall  (Total) 


Sample  Mean  Keyboard  Mean  Game  Controller 
Size  Score  (SD)  Score  (SD)  t-test  p-value 

31  11.27  (4.37)  14.13  (6.93)  -2.92  0.003 


Table  1 1 .  Matched  Paired  T-test  Comparing  Overall  (Total)  Game  Controller  to 
Scores  with  Keyboard  and  Mouse  Configuration  Scores  (Lower  is  better) 


The  participants'  scores  on  the  separate  events  were  mixed.  The 
keyboard  and  mouse  configuration  demonstrated  improved  performance  (events 
highlighted  in  yellow  bold  text  in  Table  12)  on  the  mounted  shooting, 
dismounted  moving,  and  the  shoot  house  events.  The  game  controller 
configuration  demonstrated  improved  performance  (highlighted  in  italicized  greeij 
text  in  Table  12)  on  the  mounted  moving  and  the  parking  events,  both  of  these 
events  involve  driving  a  vehicle.  The  dismounted  shooting  event  was 
inconclusive.  The  lower  the  score  demonstrates  the  better  the  performance. 
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Sample  Mean  Keyboard  Mean  Game 


Event  Size  Score  (SD)  Controller  Score  (SD)  t-test  p-value 


Mounted  Moving 

31 

3.51  (1.92) 

2.98  (1.56) 

1.73 

0.046 

Mounted  Shooting 

31 

1.77  (1.27) 

3.01  (2.37) 

-2.62 

0.007 

Dismounted  Moving 

31 

1.58  (1.09) 

2.53  (1.96) 

-2.89 

0.004 

Dismounted  Shooting 

31 

2.61  (1.17) 

2.98  (1.79) 

-0.98 

0.161 

Shoot  House 

31 

1.79  (1.15) 

2.38(1.71) 

-2.06 

0.023 

^or/f/ng 

31 

3.61  (2.39) 

2.83  (2.24) 

1.71 

0.048 

Table  12.  Matched  Paired  T-test  Comparing  Game  Controller  Scores  with  Head 
Keyboard  and  Mouse  Configuration  Scores  (Lower  is  better) 


Finally,  in  the  post  survey  16  of  the  31  (52  percent)  of  the  participants 
stated  they  preferred  the  game  controller  configuration.  The  other  15  participants 
stated  they  preferred  the  keyboard  and  mouse  configuration.  When  asked  about 
what  configuration  they  preferred  while  driving  a  vehicle  20  of  the  31  (65  percent) 
of  the  participants  stated  they  preferred  the  game  controller. 

3.  Exploratory  Hypothesis  Results 

As  an  exploratory  hypothesis,  we  examined  if  factors,  such  as  age  and 
map  representation  influence  the  effect  of  user  input  device  on  performance.  It 
would  be  valuable  to  identify  what  type  of  Soldier  could  benefit  most  from 
different  user  input  configurations  or  map  representations. 

We  first  examined  if  age  was  a  useful  contributor  when  modeling 
participant  test  performance.  Young  Solider  have  grown  up  in  a  culture  where 
video  games  are  pervasive.  It  would  be  interesting  to  see  if  younger  Soldiers 
performed  better  with  game  controllers  and  head  tracking  device  then  older 
Soldiers. 

•  Ho:  Age  contributes  nothing  useful  as  a  predictor  of  participant 
performance,  (pd  =  0) 

•  Ha:  Age  makes  a  useful  contribution  to  a  model  that  predicts 
participant  performance,  (pd  0) 
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Intercept 

Slope 

Estimate 

Estimate 

P- 

Study 

Control 

n 

(SE) 

(SE) 

F 

value 

Game  Controller 

Keyboard 

31 

12.13 

(3.22) 

-0.03  (0.13) 

0.01 

0.07 

0.785 

Game  Controller 

Game 

Controller 

31 

16.98 

(5.08) 

-0.12  (0.20) 

0.01 

0.33 

0.567 

Head  Tracker 

Head  Tracker 

22 

14.66 

(7.82) 

0.33  (0.33) 

0.04 

0.96 

0.337 

Head  Tracker 

No  Head 

Tracker 

22 

3.56(7.98) 

0.60  (0.33) 

0.13 

3.08 

0.094 

Table  13.  Inference  for  Linear  Regression  of  age  as  a  Useful  Predictor  for 

Participant  Performance 

The  overall  low  and  high  p-values  indicates  age  is  a  poor  choice  for 
building  a  model  to  predict  participant  performance.  We  do  not  reject  our  null 
hypothesis.  We  then  decided  to  explore  if  gaming  experience  could  contribute  to 
a  useful  model  to  predict  test  performance.  We  used  the  following  two  questions 
in  the  demographic  survey  to  measure  participant  gaming  experience.  1.  “How 
many  years  have  you  played  PC  video  games  on  a  regular  basis?”  and  2.  “How 
many  years  have  you  played  console  video  games  on  a  regular  basis?”  Regular 
basis  was  defined  at  least  once  a  week. 

•  Ho:  Gaming  experience  contributes  nothing  useful  as  a  predictor  of 
participant  performance,  (pd  =  0) 

•  Ha:  Gaming  experience  makes  a  useful  contribution  to  a  model  that 
predicts  participant  performance,  (pd  5^  0) 
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Intercept 

Slope 

Estimate 

Estimate 

P- 

Study 

Control 

n 

(SE) 

(SE) 

F 

value 

Game  Controller 

Keyboard 

31 

13.90 

(1.08) 

-0.25 

(0.08) 

0.33 

14.37 

0.004 

Game  Controller 

Game 

Controller 

31 

17.56 

(1.80) 

-0.32 

(0.13) 

0.17 

6.11 

0.019 

Head  Tracker 

Head  Tracker 

22 

21.19 

(2.52) 

-0.47 

(0.24) 

0.16 

3.84 

0.063 

Head  Tracker 

No  Head 

Tracker 

22 

27.92 

(2.44) 

-0.34 

(0.23) 

0.10 

2.20 

0.132 

Table  14.  Inference  for  Linear  Regression  of  Gaming  Experience  as  a  Useful 

Predictor  for  Participant  Performance 

The  values  range  from  0.10  to  0.33  indicating  that  10  percent  to  33 
percent  of  the  variation  in  the  participant  test  scores  can  be  accounted  for  by  the 
amount  of  participant's  years  of  gaming  experience.  In  the  game  controller 
study,  our  p-values  are  less  than  our  0.05  alpha.  We  reject  our  null  hypothesis 
(Ho)  in  favor  of  our  alternate  hypothesis  (Ha  ).  Based  on  the  regression  model, 
for  every  additional  year  of  gaming  experience,  we  predict  an  improvement  of 
0.25  in  performance  on  a  normalized  ten  point  scoring  system  when  using  the 
keyboard  configuration.  We  would  predict  an  improvement  of  0.32  when  using 
the  game  controller.  The  negative  slope  indicates  an  improvement  in 
performance  as  gaming  experience  increased  because  in  our  study  a  lower 
score  is  better.  Therefore,  gaming  experience  makes  a  useful  contribution  to  a 
model  predicting  participant  performance.  In  the  head  tracker  study,  gaming 
experience  p-values  are  much  lower  than  the  previous  age  p-values  but  still  are 
not  within  our  alpha  of  0.05. 

Finally,  we  wanted  to  examine  if  other  factors  such  as  sex  and  input 
device  preference  as  possible  predictors  for  test  performance.  We  found  that 
males  performed  much  better  than  female  participants.  In  the  game  controller 
study,  the  males  mean  overall  score  was  13.2  (6.2)  when  using  the  game 
controller  and  11.0  (4.4)  when  using  the  keyboard.  The  female  mean  overall 
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score  was  27.4  (0.3)  when  using  the  game  controller  and  15.3  (1.7)  when  using 
the  keyboard.  In  the  head  tracker  study,  the  males  mean  overall  score  was  21 .7 
(8.0)  when  using  the  head  tracker  and  16.6  (7.7)  when  not  using  the  head 
tracker.  The  female  mean  overall  score  was  25.3  (4.9)  when  using  the  head 
tracker  and  22.5  (11.44)  when  not  using  the  head  tracker.  We  only  had  six 
female  participants  out  of  our  53  total  participants,  so  due  to  our  small  sample 
size,  we  could  not  rely  on  sex  as  a  useful  predictor. 

We  then  wanted  to  examine  if  input  device  preference  would  make  a 
useful  contribution  to  modeling  participant  performance. 

•  Ho:  Participant  input  device  preference  contributes  nothing  useful 
as  a  predictor  of  participant  performance,  (pd  =  0) 

•  Ha:  Participant  input  device  preference  makes  a  useful  contribution 
to  a  model  that  predicts  participant  performance,  (pd  0) 


Test  Score 
when  using 

n 

Intercept 
Estimate  (SE) 

Slope 

Estimate  (SE) 

F 

p-value 

Game 

Controller 

31 

1.30(3.00) 

2.41(0.53) 

0.41 

4.53 

<0.001 

Keyboard  and 
Mouse 

31 

4.14(2.03) 

1.33  (0.36) 

0.32 

3.70 

<0.001 

Table  15.  Inference  for  linear  regression  of  participant  input  device  preference  as  a 

useful  predictor  for  participant  performance 


82 


35.00 

30.00 

o 

m  25.00 

to 

Q. 

00  20.00 
=  15.00 


8  10.00 


flj 

^  5.00 

I- 

0.00 

0  12  3  4 

Participant  Input  Device  Preference 
1  =  Keyboard,  2  =  PS3,  &  3  =  No  Pref 


♦  Total  Score  -  PS3 


Linear  (Total 
Score  -  PS3) 


Figure  19.  Graph  depicting  participant  total  scores  by  input  device 
preference  when  using  the  game  controller  configuration,  (lower 

scores  are  better) 


♦  Total  Score  -  KB 


- Linear  (Tot  a  I 

score  KB) 


Participant  Input  Device  Preference 
1  =  Keyboard,  2  =  PS3,  &  3  =  No  Pref 


Figure  20.  Graph  depicting  participant  total  scores  by  input  device 

preference  when  using  the  keyboard  and  mouse  configuration,  (lower 

scores  are  better) 
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The  R-square  values  of  0.32  to  0.41  indicating  that  32  percent  to  41 
percentof  the  variation  in  the  participant  test  scores  can  be  accounted  for  by  the 
participant's  input  device  preference.  In  both  configurations  of  our  game 
controller  study,  our  p-values  are  less  than  our  0.05  alpha.  We  were  tempted  to 
reject  our  null  hypothesis  (Ho)  in  favor  of  our  alternate  hypothesis  (Ha)  but 
thought  there  was  a  possibility  that  the  poor  performance  by  the  two  participants 
who  stated  they  had  no  user  input  device  preference  could  be  driving  our 
analysis.  Their  poor  performance  coupled  with  their  small  sample  size  of  two 
lead  us  to  recalculate  our  results  without  the  participants  who  had  no  user  input 
device  preference. 


Test  Score 
when  using 

n 

Intercept 
Estimate  (SE) 

Slope 

Estimate  (SE) 

F 

p-value 

Game 

Controller 

29 

5.57  (3.17) 

4.41(1.84) 

0.18 

5.73 

0.024 

Keyboard  and 
Mouse 

29 

4.39  (2.33) 

3.83  (1.36) 

0.23 

7.94 

0.008 

Table  16.  Linear  regression  analysis  without  participants  scores  who  had  no  user 

input  device  preference 
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Figure  21 .  Graph  depicting  participant  total  scores  by  input  device 

preference  without  participants  scores  who  had  no  user  input  device 

preference. 
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♦  Total  Score  K3 


Linear  (Ictal 
Score  -  KE) 


Participant  Input  Device  Preference 
1  =  Keyboards  2  =  PS3 


Figure  22.  Graph  depicting  participant  total  scores  by  input  device 

preference  without  participants  scores  who  had  no  user  input  device 

preference. 


The  R-square  values  of  0.18  to  0.23  were  lower  than  our  first  calculations 
but  our  p-values  were  still  less  than  our  0.05  alpha.  Again,  we  were  tempted  to 
reject  our  null  hypothesis  (Ho)  in  favor  of  our  alternate  hypothesis  (Ha)  but  we 
then  thought  the  two  possible  outliers  in  Figure  21  could  skewing  our  analysis. 
They  were  not  three  standard  deviations  from  the  mean  but  recalculate  our  game 
controller  results  for  completeness. 


Test  Score 
when  using 
Game 
Controller 

Table  17. 


Intercept  Slope 

n  Estimate  (SE)  Estimate  (SE)  F  p-value 

27  7.09(2.02)  2.88(1.19)  0.19  5.85  0.023 


Linear  regression  analysis  without  two  possible  outliers. 
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Figure  23.  Graph  depicting  participant  total  scores  by  input  device 

preference  without  two  possible  outliers. 


This  last  iteration  had  R-square  values  of  0.19  to  0.23  indicating  that  19 
percent  to  23  percent  of  the  variation  in  the  participant  test  scores  can  be 
accounted  for  by  the  participant's  input  device  preference.  Our  p-values 
continued  to  be  less  than  our  0.05  alpha.  We  reject  our  null  hypothesis  (Ho)  in 
favor  of  our  alternate  hypothesis  (Ha).  Participant  input  device  preference  makes 
a  useful  contribution  to  a  model  predicting  participant  performance.  The 
participants  that  stated  they  preferred  using  a  keyboard  and  mouse  over  a  game 
controller  performed  better  on  both  configurations.  The  participants  who  stated 
they  had  no  preference  performed  the  worst. 

The  last  factor  we  examined  was  map  representations.  Each  participant 
was  provided  a  map  and  imagery  representation  of  the  training  and  test  courses 
(See  Figures  24  and  25).  They  were  also  told  they  could  use  either  of  the  two 
representations  and  would  be  asked  for  their  preference  in  the  post  survey. 
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1.  start  Point 

2.  M4  Training  Range 

3.  M2  Training  Range 

4.  Serpentine 

5.  End  of  Training  Course 


Figure  24.  Map  Representation  of  Training  Course 


Figure  25.  Imagery  Representation  of  Training  Course 
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In  the  post  survey,  39  of  53  (75.5  percent)  participants  responded  that 
they  preferred  using  the  imagery  representation  over  the  traditional  map 
representation  when  navigating  the  training  and  test  courses. 
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VI.  RECOMMENDATION  AND  CONCLUSIONS 


A.  OVERVIEW 

Personal  computer  and  console  gaming  technology  has  advanced  to  the 
point  where  virtual  desktop  trainers  have  become  ubiquitous  to  the  U.S.  military. 
Virtual  desktop  trainers  have  the  capability  of  altering  the  interface  by  adding 
game  controllers,  larger  screens,  head  mounted  displays,  head-tracking  devices, 
and  improved  communication  interfaces.  The  military  will  gladly  pay  for 
additional  peripheral  devices  but  only  if  they  can  demonstrate  an  improvement  on 
training  effectiveness. 

The  purpose  of  this  thesis  was  to  provide  a  recommendation  and  a 
reference  to  decision  makers  when  configuring  virtual  desktop  trainer  peripheral 
interface  devices.  We  defined  the  problem  as:  “When  using  a  virtual  desktop 
simulation  for  training,  do  commercial  head-tracking  and  video  game  controller 
devices  improve  training  effectiveness  by  providing  a  more  intuitive  user 
interface?” 

B.  RESEARCH  QUESTIONS 

1.  Do  user-input  devices  impact  military  training  performance  when 
used  with  a  desktop  virtual  training  simulation? 

2.  Do  soldiers  prefer  one  user-input  device  over  another  when  using  a 
desktop  virtual  training  simulation? 

C.  OVERALL  ASSESSMENT 

Our  overall  assessment  of  study  is  very  positive.  The  significant  time  and 
effort  in  preparing,  executing,  and  analyzing  the  pilot  study,  enable  us  to  employ 
a  final  within  subjects  experiment  capable  of  producing  quality  data.  We  have 
the  upmost  confidence  in  our  data  and  our  analysis.  We  found  that  the  keyboard 
and  mouse  was  superior  to  the  game  controller  configuration  in  overall 
performance.  The  one  exception  was  during  the  driving  event  where  the  game 
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controller  was  superior.  The  head  tracker  was  found  to  be  detrimental  to  overall 
performance.  When  training  with  desktop  virtual  simulations,  we  recommend  a 
user  input  configuration  consisting  of  the  keyboard  and  mouse  without  the  head 
tracker  as  a  baseline  configuration.  We  also  recommend  implementing  game 
controllers  as  an  additional  input  device  for  units  training  tasks  that  require 
trainees  to  drive  vehicles.  We  believe  this  configuration  provides  a  more  intuitive 
user  interface  that  minimizes  the  initial  training  time  required  to  teach  users  to 
perform  basic  individual,  weapon,  and  vehicle  functions  in  a  virtual  desktop 
simulator.  This  time  savings  should  result  in  more  time  available  to  train  the 
skills  that  will  keep  a  soldier  alive  in  combat  instead  to  learning  how  to  interact 
with  the  simulation. 

D.  DISCUSSION  OF  RESULTS 

1.  Head  Tracker  Study 

Contrary  to  our  hypothesis,  use  of  the  head  tracker  did  not  improve  overall 
performance.  Based  on  this  result,  we  were  surprised  that  50  percent  of 
participants  reported  a  favorable  head  tracker  preference.  When  participants 
donned  the  TracklR5™  baseball  caps  and  first  used  the  head  tracker,  the 
excitement  was  obvious.  Initial  comments  ranged  from  “cool,”  “awesome,”  to 
“greatest  thing  ever.”  At  the  end  of  the  experiment,  participants  were  less  excited 
but  mostly  voiced  their  support  of  the  head  tracker.  In  retrospect,  we  believe  the 
participants  continued  to  voice  their  support  of  the  head  tracker  at  the  end  of  the 
experiment  because  they  believed  it  was  what  we  wanted  to  hear. 

We  assumed  in  the  formulation  of  our  hypothesis,  at  worst,  the  head 
tracker  would  have  no  effect  on  performance.  This  was  wrong.  We  actually 
found  that  the  head  tracker  had  an  adverse  effect  on  overall  performance. 
Additionally,  the  head  tracker  had  an  adverse  effect  on  four  of  the  six  events  and 
was  inconclusive  on  the  other  two  events.  Despite  this  adverse  affect  on 
performance,  50  percent  of  the  participants  stated  they  preferred  the  addition  of 
the  head  tracker. 
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The  inability  of  the  head  tracker  to  improve  performance  was  not  a 
surprise.  We  were  surprised  when  we  found  the  head  tracker  actually  had  an 
adverse  effect  on  performance.  The  head  tracker  does  provide  an  additional 
amount  of  control  to  the  participant.  This  additional  amount  of  control  is  small 
and  comes  at  a  cost.  This  cost  comes  in  the  complexity  of  an  additional  input 
device.  A  participant  with  more  advanced  gaming  skills  might  appreciate  and 
exploit  the  additional  control  but  most  found  it  overwhelming.  When  participants 
inadvertently  move  the  head  tracker,  they  unknowingly  changed  the  player 
perspective  in  the  virtual  world.  At  first,  most  participants  were  using  the  head 
tracker  but  by  the  end  of  the  test,  most  participants  kept  their  head  as  still  as 
possible.  Keeping  their  heads  locked  in  one  position  essentially  disabled  the 
head  tracker.  This  limited  their  ability  to  view  the  surrounding  area  but  kept  their 
viewing  perspective  consistent.  By  keeping  their  view  perspective  consistent,  the 
participant  improved  their  ability  to  control  their  player  and  limited  the  possibly  of 
disorientation  and  getting  lost  in  the  virtual  environment. 

The  50  percent  of  participants  who  stated  they  preferred  the  head  tracker 
despite  performing  better  without  the  head  tracker  was  surprising.  Four  of  the 
twenty-two  participants  actually  performed  better  with  the  head  tracker  and  it  is 
logical  they  would  prefer  the  head  tracker  configuration.  The  inconsistency  lies 
with  the  seven  participants  who  performed  poorly  with  the  head  tracker  but  stated 
they  preferred  it.  We  believe  the  reason  for  this  inconsistency  is  a  combination  of 
the  newness  of  the  technology  and  limiting  participants  to  only  20  minutes  of 
training  prior  to  testing.  It  is  possible  these  seven  participants  saw  the  potential 
of  this  technology  and  believed  if  they  were  given  more  time  using  the  head 
tracker  they  would  have  performed  better.  The  participants  may  have  been 
sending  us  a  message  that  more  time  was  needed  to  attain  an  acceptable  level 
of  proficiency  using  the  head  tracker.  Conversely,  these  results  indicate  that 
subjective  preference  may  not  be  a  good  indicator  of  actual  performance. 
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2.  Game  Controller  Study 

Our  literature  review  indicated  that  people  had  strong  opinions  about 
game  controllers,  but  the  opinions  were  based  on  personal  preference  and  not 
reason.  The  pilot  study  results  demonstrated  an  improved  performance  with  the 
game  controller  but  we  believed  that  sequence  was  the  determining  factor — not 
input  device.  In  the  pilot  study,  all  participants  tested  with  the  keyboard  and 
mouse  configuration  first  and  then  the  game  controller  second.  This  learning 
effect  was  considered  in  our  experimental  design  and  balanced  within  our  final 
study. 

After  participants  completed  the  DLI  experiment,  we  conducted  verbal  exit 
polling  that  provided  no  insights  on  potential  results.  Participant  comments 
equally  praised  and  disparaged  both  configurations.  This  made  the  data  analysis 
that  much  more  exciting  because  we  truly  did  not  know  what  to  expect. 

Contrary  to  our  hypothesis,  use  of  the  game  controller  did  not  improve 
overall  performance  and  the  keyboard  and  mouse  configuration  was  superior. 
Based  on  this  result,  we  were  surprised  that  52  percent  of  participants  reported  a 
favorable  game  controller  preference.  The  keyboard  and  mouse  was  superior  in 
three  of  the  six  events,  the  game  controller  was  superior  in  two  events,  and  one 
event  was  inconclusive.  The  game  controller  was  superior  in  the  two  driving 
events. 

The  keyboard  and  mouse  outperformed  the  game  controller  in  the 
mounted  shooting,  dismounted  moving,  and  shoot  house  events.  We  believe  the 
reason  for  this  has  less  to  do  with  the  keyboard  and  more  to  do  with  the  mouse. 
The  mouse  is  designed  for  micro  movements.  The  limiting  factor  in  the  shooting 
events  is  the  exact  placement  of  the  reticule  on  the  target  or  positioning  of  the 
player.  When  using  the  mouse  to  sight  a  weapons  or  position  a  player,  the 
participant  uses  his  forearm,  wrist,  and  fingers  to  move  the  mouse  to  the  desired 
location.  When  using  the  game  controller,  only  the  thumb  is  used  to  sight  the 
weapon  or  position  the  player.  The  additional  number  of  muscles  used  with  the 
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mouse  increases  the  participant's  ability  to  precisely  control  the  weapon  or 
player.  Therefore,  when  precision  is  required,  the  keyboard  and  mouse  proved 
superior  to  the  game  controller. 

The  game  controller  outperformed  the  keyboard  and  mouse  in  the  two 
driving  events.  In  these  events,  precision  was  not  the  key  factor  in  performance. 
We  believe  the  motivating  factor  was  simplicity.  When  using  the  keyboard  and 
mouse  to  drive  the  vehicle,  the  participant  used  the  W  key  to  move  forward  and 
the  A  and  D  keys  to  move  left  and  right,  respectively.  When  the  participant 
pressed  the  A  key  the  vehicle  made  a  full  90  degree  turn  and  when  they  released 
the  key,  the  steering  wheel  returned  to  the  center  position.  This  resulted  in  an  all 
or  nothing  effect.  If  the  participant  wanted  to  turn  45  degrees,  then  they  would 
have  to  alternate  pressing  the  A  and  the  W  keys  back  and  forth.  When  using  the 
game  controller,  the  participants  simply  move  the  thumb  stick  in  the  direction  of 
the  turn  and  the  vehicle  responded.  The  mouse  could  also  be  used  to  turn  the 
vehicle  but  also  proved  to  be  very  difficult  to  control.  The  vehicle  would  turn  as 
long  as  the  mouse  was  moving  to  the  left  or  the  right,  but  when  the  mouse  ran  off 
the  mouse  pad,  the  table,  or  beyond  the  reach  of  the  participant  the  steering 
wheel  would  return  to  the  center  position. 

The  game  controller  was  also  superior  in  controlling  the  speed  of  the 
vehicle.  When  the  participant  pressed  the  W  key  to  move  forward  the  HUMMV 
accelerated  to  top  speed  and  rolled  to  a  stop  when  released.  Again,  it  was  an  all 
or  nothing.  This  resulted  in  participants  tapping  on  the  W  key  to  correspond  with 
how  fast  they  wanted  to  move.  When  the  participant  used  the  game  controller 
and  wanted  to  go  slow,  they  would  slightly  move  the  thumb  controller  forward 
and  the  vehicle  would  respond.  If  they  wanted  to  increase  the  speed,  they 
increased  pressure  on  the  thumb  control.  The  simplicity  of  the  game  controller 
over  the  keyboard  and  mouse  was  the  determining  factor  in  the  driving  events. 

We  were  surprised  when  52  percent  of  participants  stated  they  preferred 

the  game  controller  despite  performing  testing  indicated  the  keyboard  and  mouse 

was  the  superior  configuration.  Nine  of  the  thirty-one  participants  actually 
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performed  better  with  the  game  controller  and  it  is  logical  they  would  prefer  the 
game  controller  configuration.  The  inconsistency  lies  with  the  seven  participants 
who  performed  poorly  with  the  game  controller  but  stated  they  preferred  it.  We 
believe  the  reason  for  this  inconsistency  is  the  perception  between  personal 
computers  and  console  game  systems.  The  prevailing  perception  is  personal 
computers  are  for  work  and  console  game  systems  are  for  play.  We  believe  this 
perception  between  work  and  play  is  what  lead  to  the  majority  of  participants 
stating  they  preferred  the  game  controller  despite  performing  better  with  the 
keyboard  and  mouse  configuration. 

3.  Exploratory  Hypothesis  Results 

Contrary  to  our  hypothesis,  age  was  not  a  useful  contributor  when 
modeling  participant  test  performance.  Young  soldiers  have  grown  up  in  a 
culture  where  video  games  are  pervasive  and  we  thought  this  would  provide 
them  an  advantage  over  older  Soldiers.  Our  data  did  not  support  this  hypothesis. 
Alternatively,  the  age  range  in  our  sample  was  not  large  enough  to  capture  age 
differences.  Next,  we  tested  a  second  exploratory  hypothesis  to  find  out  whether 
gaming  experience  would  contribute  to  a  useful  model  to  predict  test 
performance.  We  found  that  gaming  experience  in  fact  did  contribute  to  a  useful 
model  to  predict  performance.  These  findings  are  logical.  A  Soldier's  youth  does 
not  automatically  translate  to  video  game  skills,  but  years  of  gaming  experience, 
regardless  of  age,  does  translate  to  superior  performance. 

We  then  tested  a  third  hypothesis  to  find  out  whether  participant  input 
device  preferences  make  a  useful  contribution  to  a  model  that  predicts  participant 
performance.  Consistent  with  our  hypothesis,  input  device  preference  did  predict 
performance.  We  originally  thought  that  participants  would  be  able  to  predict 
what  input  device  they  would  perform  best  with,  but  they  did  not.  We  found  that 
participants  who  preferred  the  keyboard  and  mouse  configuration  performed 
better  on  both  user  input  configurations.  Those  who  preferred  the  game 
controllers  had  worse  performance  scores.  We  believe  this  reinforces  our 
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findings  that  the  keyboard  and  mouse  is  a  superior  input  device.  The  participants 
who  achieved  the  best  performance  scores  were  also  those  with  several  years  of 
gaming  experience  and  presumably  knew  before  the  experiment  that  the 
keyboard  and  mouse  was  a  superior  input  device.  These  participants  did  not 
need  the  results  of  a  study  to  tell  them  that  the  keyboard  and  mouse  was 
superior  to  the  game  controller.  They  had  years  of  gaming  experience,  playing 
all  types  of  video  games,  to  tell  them  the  keyboard  and  mouse  was  superior. 
This  may  call  into  question  why  did  we  conduct  an  experiment  instead  of  a 
survey.  The  challenge  in  conducting  such  a  survey  would  be  who  do  you  poll? 
We  only  found  out  who  the  experts  were  after  the  experiment.  Prior  to  the 
experiment,  everyone  just  had  opinions  but  after  the  experiment  we  could  identify 
who  the  experts  were  by  their  superior  performance  scores.  When  we  looked  at 
the  experts  input  device  preference  they  were  over  overwhelming  in  support  of 
the  keyboard  and  mouse  configuration.  We  believe  this  provides  further  support 
to  our  findings  that  the  keyboard  and  mouse  is  a  superior  user  input  device  when 
training  with  desktop  virtual  simulations. 

The  last  factor  we  examined  was  map  representations.  In  the  post  survey, 
75.5  percent  of  participants  responded  that  they  preferred  using  the  imagery 
representation  over  the  traditional  map  representation  when  navigating  the 
training  and  test  courses.  We  thought  there  may  be  a  relationship  between  age 
and  map  representation.  We  hypothesized  that  younger  Soldiers  would  prefer 
imagery  and  older  soldiers  would  prefer  traditional  maps.  We  could  not  support 
this  hypothesis  and  found  most  soldiers  preferred  the  imagery  regardless  of  age. 
We  believe  this  finding  was  the  result  of  the  additional  information  provided  by 
imagery.  Both  the  imagery  and  traditional  map  provided  sufficient  information  to 
navigate  the  course.  The  imagery  simply  provided  more  information  to  the 
participant.  One  example  is  the  traditional  map  simply  provided  the  location  of  a 
building  and  all  of  the  buildings  looked  the  same.  The  imagery  provided  the 
location,  color,  and  the  relative  height  of  the  buildings.  Furthermore,  the  maps 
representation  of  terrain  elevation  and  vegetation  was  very  poor  as  compared  to 
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what  the  detail  provided  by  the  imagery.  Lastly,  the  map  did  not  represent  details 
such  as  highway  signs  that  were  clearly  represented  in  the  imagery.  All  of  this 
additional  information  resulted  in  the  imagery  being  favored  over  the  traditional 
maps. 

E.  LIMITATIONS 

1.  Time 

The  biggest  limitation  in  our  experiment  was  time.  Specifically,  the  time 
participants  were  allowed  to  train  on  the  user  input  device  prior  to  testing.  We 
limited  this  training  time  to  20  minutes.  Our  results  could  have  been  far  different  if 
we  had  doubled  or  tripled  the  training  time. 

2.  Training  Task 

We  investigated  the  effects  of  head  trackers  and  game  controllers  on 
essentially  four  training  task.  These  tasks  were  driving  a  HUMMV,  shooting  a  .50 
Cal  machinegun,  dismounted  movement,  and  shooting  a  M4  rifle.  The  number  of 
task  VBS2™  can  train  is  only  limited  by  the  imagination  of  the  trainer.  It  would  be 
interesting  to  see  our  study  expanded  into  different  training  task  such  as  tank 
crew,  aviation  crew,  call  for  fire,  and  reconnaissance  tasks. 

3.  Sample  Population 

The  participants  for  this  study  were  all  U.S.  Army  operational  support 
Soldiers  enrolled  in  language  training  at  the  Defense  Language  Institute  (DLI). 
There  is  a  important  difference  between  combat  arms  Soldiers  and  a  operational 
support  Soldiers.  Our  participant  population  was  chosen  because  they  were 
young  Soldiers  but  limited  because  they  were  all  operational  support  Soldiers 
and  no  combat  arms  Soldiers  were  included  due  to  availability. 

A  second  sample  population  limitation  was  the  number  of  females 
participants.  We  found  that  males  performed  much  better  than  female 
participants  on  each  configuration.  We  only  had  six  female  participants  out  of 
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our  53  total  participants.  This  small  sample  size  limited  our  ability  to  find 
statistical  significance  within  our  female  sample  population. 

F.  RECOMMENDATIONS 

1.  Configuration  Recommendations 

When  training  with  desktop  virtual  simulations,  we  recommend  a  user 
input  configuration  consisting  of  the  keyboard  and  mouse  without  the  head 
tracker  as  a  baseline.  We  also  recommend  game  controllers  as  an  additional 
input  device  for  when  units  are  training  tasks  that  require  trainees  to  drive 
vehicles.  For  example,  if  a  unit  is  training  HUMMV  convoy  operations  with  a 
traditional  4  man  vehicle  crew,  we  would  recommend  all  four  crew  members 
have  the  baseline  configuration  with  the  addition  of  one  game  controller  for  the 
vehicle  driver. 

2.  Recommendation  for  Future  Work 

a.  Testing  of  Other  User  Input  Devices 

We  only  tested  two  different  user  input  devices.  Virtual  desktop 
trainers  have  the  capability  of  altering  the  interface  by  adding  various  game 
controllers,  steering  wheels,  and  improved  communication  interfaces. 
Investigating  the  any  of  these  user  input  devices  could  result  in  improved  training 
effectiveness  would  be  a  worthy  endeavor. 

b.  Testing  of  Different  Visual  Displays 

We  conducted  our  test  using  the  standard  laptop  screen  provided 
by  the  DVTE  suite.  It  would  be  valuable  to  find  out  what  visual  deliver  system 
best  creates  the  immersive  effect  and  how  much  field  of  view  does  it  require. 
The  controls  of  interest  are  laptop  screens,  larger  monitors,  wall  sized  project 
screen,  panoramic  displays,  and  head  mounted  displays. 
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c.  Integrating  Physical  Active  With  Virtual  Training 

Our  participants  sat  in  front  of  a  laptop  computer  as  they  drove,  ran, 


and  shot  their  way  through  a  challenging  evaluation  course.  How  would 
integrating  a  virtual  sphere,  carrying  a  wireless  rifle,  and  wearing  a  head 
mounted  display  impact  training  effectiveness  when  used  in  conjunction  with  a 
desktop  virtual  simulation  such  as  VBS2™. 

d.  Change  Sample  Population 

The  participants  for  this  study  were  all  U.S.  Army  operational 
support  Soldiers  enrolled  in  language  training  at  the  Defense  Language  Institute 
(DLI).  There  is  a  significant  difference  between  a  combat  arms  Soldier  and  a 
operational  support  soldiers.  How  would  combat  arms  Soldiers'  performance 
compare  to  operational  support  Soldiers  performance  in  the  same  test. 

We  only  had  six  female  participants  out  of  our  53  total  participants. 
This  small  sample  size  limited  our  ability  to  find  statistical  significance  within  our 
female  sample  population.  It  would  be  valuable  to  know  if  there  is  a  gender 
difference  when  training  with  desktop  virtual  simulations. 
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APPENDIX  A.  PILOT  STUDY  COMPUTER  SET  UP 


1)  Unload  VBS2  Boxes 

•  Each  individual:  Laptop,  power  cords,  mouse,  and  LAN  Cable 

•  Each  row:  Switch,  LAN  Cable,  and  Surge  suppressor 

2)  Start  Brief 

•  Intro 

•  Agenda 

•  Background 

•  Paperwork 

3)  Read  and  sign  IRB  Consent 

4)  Collect  IRB  Consent 

5)  Turn  On 

6)  Login 

•  Ctrl-Alt-Del  x  2 

•  User  Name:  administrator 

•  Password:  DVTEM90build2 

•  OK 

7)  Set  Time  on  Clock 

8)  Access  Shared  Folder  on  “DVTE30” 

•  Right  Click  “Start” 

•  Explore 

•  Scroll  to  the  bottom  and  open  “My  network  places/Entire 
Network/Microsoft  Windows  Network/MShome/DVTE30 

•  Click  to  open  SharedDocs  Folder 

•  Copy  and  paste  “Pilot  Study  Download”  folder  into  your 
“Administrator/My  Documents”  folder 

9)  Profile 

•  Open  “Pilot  Study  Download\VBS2  input  profiles” 

•  Copy  “AdministratorKBNew.VBSProfile”  (file) 

•  Paste  in  C:\Documents  and  Settings\administrator\My 
Documents\VBS2 

•  Rename  current  “Administrator.VBSProfile”  to 
“AdministratorOld.VBSProfile” 
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•  Rename  “Administrator  KBNew.VBSProfile”  to 
“Administrator.VBS2Profile” 

10)  Interface  Scenario 

•  Open  “Pilot  Study  Download” 

•  Copy  “InterfaceTrainerPilot.Sara”  folder 

•  Paste  in  C:\Documents  and  Settings\administrator\My 
Documents\VBS2\MPMissions 

11)  Interface  Lib-Interface  Data 

•  Open  “Pilot  Study  Download” 

•  Copy  “InterfaceLib”  folder 

•  Paste  in  CAProgram  Files\Bohemia  lnteractive\VBS2  VTK\  (New 
place) 

•  Open  “Pilot  Study  Download\Jar  files” 

•  Copy  the  “InterfaceData##”  file  that  corresponds  with  your  VBS2  laptop 
number  (113-  DVTE-##) 

•  Paste  in  CAProgram  Files\Bohemia  lnteractive\VBS2  VTKMnterfaceLib 

12)  Interface  batch  file 

•  Open  “Pilot  Study  DownloadMnterfaceDataCollectorBat” 

•  Copy  the  “InterfaceDataCollector##”  file  that  corresponds  with  your 
VBS2  laptop  number  (1 1 3  -  DVTE  -  ##) 

•  Paste  in  CAProgram  Files\Bohemia  lnteractive\VBS2  VTK\ 

13)  disEntity  and  VBSEntity  Config  files 

•  Open  “Pilot  Study  DownloadV’ 

•  Copy  “disEntity. config”  and  “vbsEntity.config”  files 

•  Paste  in  CAProgram  Files\Bohemia  lnteractive\VBS2  VTK\config 

•  Answer  yes  to  all  “overwrite” 

14)  Modify  DIS  Deploy  file. 

•  Open  CAProgram  Files\Bohemia  lnteractive\VBS2  VTK\config 

•  Right  click  on  the  “dis. deploy”  file 

•  Left  click  on  “edit” 

•  The  7th  line  reads 

“<broadcastAddress>10.1.10.###</broadcastAddress>“ 

•  Change  line  to  read 

“<broadcastAddress>10.1 .10.255</broadcastAddress>“ 

•  The  8th  line  reads  “<broadcastPort>10000</broadcastPort> 

•  Change  line  to  read  “<broadcastPort>100##</broadcastPort>,”  where 
##  is  your  VBS2  laptop  number 

•  Save  file  with  same  name 

•  Close  file 
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15)  PS3  Controller  and  Headtracker  driver 

•  Open  “Pilot  Study  Download\Drivers” 

•  Double  click  on  the  TracklR_4.1 .037. Final  - 

•  Follow  instructions  - 

o  Ok 
o  Next 
o  Yes 
o  Next 
o  Next 
o  Next 
o  Next 

o  No  (when  asked  to  scan  for  old  INF  files) 
o  Finish 

•  Open  “Pilot  Study  Download\Drivers”  and  Double  click  on  “wrar390” 

•  Install 

•  When  asked  to  associate  with?  chose  “RAR” 

•  Click  “Done” 

•  Close  win  War  Screen 

•  Open  “Pilot  Study  Download\Drivers” 

•  Double  click  on  1167_Sixaxis_PS3_Win32 

•  Click  close 

•  Click  on  “Extract  to” 

•  On  right  hand  side  chose  to  extract  to  desktop 

•  Close  window 

•  Create  a  folder  on  desktop  called  PS3 

•  Move  the  three  new  file  into  the  folder  called  PS3 

o  (libusb-win,  libusb0.dll,  and  ps3sixaxis) 

16)  Emergency  post  survey 

•  Create  folder  on  desktop  called  “Emergency  Post  Survey” 

•  Open  “Pilot  Study  Download\Jar  Files” 

•  Copy  “EmergegcyPostSurvey_9999 

•  Paste  in  desktop\  Emergency  Post  Survey 

17)  Shortcuts 

•  Open  “  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK” 

•  Right  click  on  “VBS2  LVC  Game” 

•  Hover  over  “Send  to” 

•  Select  “Desktop  (create  shortcut)” 

•  Right  click  on  “InterfaceDataCollector##” 

•  Hover  over  “Send  to” 

•  Select  “Desktop  Create  shortcut)” 
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Rename  the  short  cut  by  removing  the  “Shortcut  to’ 


18)  Right  click  the  desktop  and  arrange  icons  by  type. 

1 9)  Check  your  work 

•  Double  click  on  “VBS2  LVC  Game”  Short  Cut  -  Wait  awhile 

•  Click  on  networking 

•  Click  on  new 

•  Click  on  Sahrani 

•  Click  on  InterfaceTrainerPilot 

•  Click  on  OK 

•  Click  on  Marine  (bottom  left) 

•  Click  on  OK 

•  Wait 

•  Click  continue 

•  After  the  scenario  loads  press  “ALT  &  Tab” 

•  Double  click  InterfaceDataCollector##  Shortcut 

•  Enter  0001  as  your  participant  number 

•  Complete  the  survey  and  wait  for  4-5  seconds 

•  Your  DOS  window  should  then  read 

“All  trigger  catalogued” 

“All  targets  catalogued” 

If  your  window  provides  this  message  then  you  get  a  “A”  for  this 
lesson. 

If  not,  additional  instruction  will  be  provided  this  Saturday  at  1200. 

20)  Control-C 

21)  Yes  and  enter 

22)  Click  on  VBS2 

23)  Escape 

24)  Abort 

25)  OK 

26)  Escape 

27)  Cancel 

28)  Cancel 

29)  Cancel 

30)  Big  red  X 

31 )  Close  windows  explorer 
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APPENDIX  B.  PILOT  STUDY  INFORMED  CONSENT 


Informed  Consent  Form 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  “Impact  of 
User  Interfaces  on  Virtual  Battlespace  2  Training  Pilot  Study.”  This  study  supports 
a  project  to  compare  different  user-input  interfaces  and  visual  peripherals  devices 
when  conducting  military  training  using  VBS2™. 

This  pilot  study  tests  procedures  to  familiarize  users  with  the  simulation. 

Procedures.  The  pilot  study  will  consist  of  the  following: 

-  Survey  to  better  understand  the  user’s  level  of  computer 
expertise; 

-  Overview  brief  describing  the  project  and  the  pilot  study’s 
purpose; 

-  User  interface  brief  describing  the  basic  functions  of  the 
simulation; 

-  User  training,  where  participants  will  be  able  to  move  dismount 
and  drive  a  HUMMV.  The  participants  will  also  engage  targets 
with  personnel  weapons  and  mounted  crew  served  weapons  in 
the  VBS2™  simulation; 

-  User  evaluation,  where  participants  will  complete  a  short 
obstacle  course  demonstrating  their  skill  level  with  the 
simulation. 

The  pilot  study  will  take  no  longer  than  the  50  minute  class  period. 

Risks.  The  potential  risks  of  participating  in  this  study  are  not  greater  than 
minimal  risk.  The  study  involves  no  known  reasonably  foreseeable  risks  or 
hazards  greater  than  those  encountered  in  everyday  life. 

Benefits.  The  anticipated  benefit  from  this  study  is  gaining  insight  into  how  to  best 
configure  first  person  shooter  simulations  when  using  them  as  tactical  training 
devices. 

Compensation.  No  tangible  compensation  will  be  given.  A  copy  of  the  research 
results  will  be  available  at  the  conclusion  of  the  experiment.  If  you  would  like  a 
copy  of  the  results,  e-mail  Major  William  Glaser  at  wrglaser@nps.edu. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this 
study  will  be  kept  confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within 
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reason,  will  be  made  to  keep  your  personal  information  in  your  research  record 
confidential  but  total  confidentiality  cannot  be  guaranteed.  No  information  that 
could  identify  a  participant  will  be  publicly  accessible.  Records  of  participation 
will  be  maintained  by  NPS  for  3  years,  after  which  they  will  be  destroyed. 
However,  it  is  possible  that  the  researcher  may  be  required  to  divulge  information 
obtained  in  the  course  of  this  research  to  the  subject’s  chain  of  command  or 
other  legal  body. 

Voluntary  Nature  of  the  Study.  Participation  in  this  study  is  strictly  voluntary,  and 
if  agreement  to  participation  is  given,  it  can  be  withdrawn  at  any  time  without 
prejudice. 

Points  of  Contact.  It  is  understood  that  should  any  questions  or  comments  arise 
regarding  this  project,  or  a  research  related  injury  is  received,  the  Principal 
Investigator,  Dr.  William  J.  Becker,  656-3963,  wibecker@nps.edu  should  be 
contacted.  Any  other  questions  or  concerns  may  be  addressed  to  the  Navy 
Postgraduate  School.  IRB  Chair,  LCDR  Paul  O’Connor  ,  831-656-3864, 
peoconno@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been 
given  the  opportunity  to  ask  questions  and  all  the  questions  have  been  answered 
to  my  satisfaction.  I  have  been  provided  a  copy  of  this  form  for  my  records  and  I 
agree  to  participate  in  this  study.  I  understand  that  by  agreeing  to  participate  in 
this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature  Date 


WILLIAM  R.  GLASER 
MAJ,  AR  (FA57) 

NPS  MOVES  Student 
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APPENDIX  C.  PILOT  STUDY  APPROVED  IRB  PROTOCOL 


Naval  Postgraduate  School 

Institutional  Review  Board  (IRB) 


From;  President,  Naval  Postgraduate  School  2  5  2010 

Via:  Chairman,  Institutional  Review  Board 

To;  William  Becker,  MOVES  Institute 

William  Glaser 


Subject:  PROJECT  STATUS:  IMPACT  OF  USER  INTERFACES  ON  VIRTUAL 

BATTLESPACE  2  TRAINING  PILOT  STUDY 


1 .  The  NPS  IRB  is  pleased  to  inform  you  that  the  NPS  Institutional  Review 
Board  has  approved  your  project  (WS  IRB#  NPS.2010.0023-IR-EP7-A). 
Completion  of  the  CITI  Research  Ethics  Training  has  been  confirmed. 

2,  This  approval  is  valid  until  1  August  10.  Please  submit  all  reporting 
requirements  to  the  Human  Subjects  Office  (Rikki  Panis,  Halligan  Hall, 
Room  283)  at  the  conclusion  of  this  project. 

3.  If  your  protocol  changes  at  any  time,  you  will  need  to  resubmit  your 
project  proposal  to  the  NPS  IRB. 

4,  As  the  Principal  Investigator  it  is  your  responsibility  to  ensure  that  the 
research  and  the  actions  of  all  project  personnel  involved  in  conducting 
this  study  will  conform  with  the  IRB  approved  protocol  and  IRB 
requireraents/policies. 


5. 


After  the  experiment  is  completed  the  Principal  Investigator  will  submit  to 
the  Human  Subjects  Research  Office,  all  signed  informed  consent 
documents,  unanticipated  problem  reports,  adverse  event  reports  and  a 
final  report.  The  Human  Subjects  Research  Office  will  secure  these 
documents  for  three  (3)  years  and  then  destroy,  as  prescribed  by  law. 


Angela  O^Dea  PhD 
Chair 

Institutional  Review  Board 


Daniel  T.  Oliver 
President 

Naval  Postgraduate  School 
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APPENDIX  D.  DU  STUDY  COMPUTER  SET  UP 


1)  Turn  computer  on 

2)  Login 

•  Ctrl-Alt-Del  x  2 

•  User  Name:  administrator 

•  Password:  DVTEM90build2 

•  OK 

3)  Access  Shared  Folder  on  “DVTE30” 

•  Right  Click  “Start” 

•  Explore 

•  Scroll  to  the  bottom  and  open  “My  network  places/Entire 
Network/Microsoft  Windows  Network/MShome/DVTE30 

•  Click  to  open  SharedDocs  Folder 

•  Copy  and  paste  “Pilot  Study  Download”  folder  into  your  “Administrator/My 
Documents”  folder 

4)  Profile 

•  Open  “Pilot  Study  Download\VBS2  input  profiles” 

•  Copy  “AdministratorKBNew.VBSProfile”  (file) 

•  Paste  in  C:\Documents  and  Settings\administrator\My  Documents\VBS2 

•  Rename  current  “Administrator.VBSProfile”  to 
“AdministratorOld.VBSProfile” 

•  Rename  “Administrator  KBNew.VBSProfile”  to  “Administrator.VBS2Profile” 

5)  Interface  Scenario 

•  Open  “Pilot  Study  Download” 

•  Copy  “InterfaceTrainerPilot.Sara”  folder 

•  Paste  in  C:\Documents  and 
Settings\administrator\Mydocuments\VBS2\MPMissions 

6)  Interface  Lib-Interface  Data 

•  Open  “Pilot  Study  Download” 

•  Copy  “InterfaceLib”  folder 

•  Paste  in  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK\  (New  place) 

•  Open  “Pilot  Study  Download\Jar  files” 

•  Copy  the  “InterfaceData##”  file  that  corresponds  with  your  VBS2  laptop 
number  (113-  DVTE-##) 

•  Paste  in  C:\Program  Files\Bohemia  lnteractive\VBS2  VTKMnterfaceLib 


7)  Interface  batch  file 
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•  Open  “Pilot  Study  DownloadMnterfaceDataCollectorBat” 

•  Copy  the  “InterfaceDataCollector##”  file  that  corresponds  with  your  VBS2 
laptop  number  (113-DVTE-##) 

•  Paste  in  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK\ 

8)  disEntity  and  VBSEntity  Config  files 

•  Open  “Pilot  Study  DownloadV’ 

•  Copy  “disEntity. config”  and  “vbsEntity.config”  files 

•  Paste  in  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK\config 

•  Answer  yes  to  all  “overwrite” 

9)  Modify  DIS  Deploy  file. 

•  Open  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK\config 

•  Right  click  on  the  “dis. deploy”  file 

•  Left  click  on  “edit” 

•  The  7th  line  reads  “<broadcastAddress>10.1 .10.###</broadcastAddress>“ 

•  Change  line  to  read 

“<broadcastAddress>10.1 .10.255</broadcastAddress>“ 

•  The  8th  line  reads  “<broadcastPort>10000</broadcastPort> 

•  Change  line  to  read  “<broadcastPort>100##</broadcastPort>,”  where  ## 
is  your  VBS2  laptop  number 

•  Save  file  with  same  name 

•  Close  file 

10)  PS3  Controller  and  Fleadtracker  driver 

•  Open  “Pilot  Study  Download\Drivers” 

•  Double  click  on  the  TracklR_4.1 .037. Final  - 

•  Follow  instructions  - 

(a)  Ok 

(b)  Next 

(c)  Yes 

(d)  Next 

(e)  Next 

(f)  Next 

(g)  Next 

(h)  No  (when  asked  to  scan  for  old  INF  files) 

(i)  Finish 

•  Open  “Pilot  Study  Download\Drivers”  and  Double  click  on  “wrar390” 

•  Install 

•  When  asked  to  associate  with?  chose  “RAR” 

•  Click  “Done” 

•  Close  win  War  Screen 

•  Open  “Pilot  Study  Download\Drivers” 
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•  Double  click  on  1167_Sixaxis_PS3_Win32 

•  Click  close 

•  Click  on  “Extract  to” 

•  On  right  hand  side  chose  to  extract  to  desktop 

•  Close  window 

•  Create  a  folder  on  desktop  called  PS3 

•  Move  the  three  new  file  into  the  folder  called  PS3  (libusb-win,  libusb0.dll, 
and  ps3sixaxis) 

1 1 )  Emergency  post  survey 

•  Create  folder  on  desktop  called  “Emergency  Post  Survey” 

•  Open  “Pilot  Study  Download\Jar  Files” 

•  Copy  “EmergegcyPostSurvey_9999 

•  Paste  in  desktop\  Emergency  Post  Survey 

12)  Shortcuts 

•  Open  “  C:\Program  Files\Bohemia  lnteractive\VBS2  VTK” 

•  Right  click  on  “VBS2  LVC  Game” 

•  Hover  over  “Send  to” 

•  Select  “Desktop  (create  shortcut)” 

•  Right  click  on  “InterfaceDataCollector##” 

•  Hover  over  “Send  to” 

•  Select  “Desktop  Create  shortcut)” 

•  Rename  the  short  cut  by  removing  the  “Shortcut  to” 

13)  Right  click  the  desktop  and  arrange  icons  by  type. 

14)  Check  your  work 

•  Double  click  on  “VBS2  LVC  Game”  Short  Cut  -  Wait  awhile 

•  Click  on  networking 

•  Click  on  new 

•  Click  on  Sahrani 

•  Click  on  InterfaceTrainerPilot 

•  Click  on  OK 

•  Click  on  Marine  (bottom  left) 

•  Click  on  OK 

•  Wait 

•  Click  continue 

•  After  the  scenario  loads  press  “ALT  &  Tab” 

•  Double  click  InterfaceDataCollector##  Shortcut 

•  Enter  0001  as  your  participant  number 

•  Complete  the  survey  and  wait  for  4-5  seconds 

•  Your  DOS  window  should  then  read 
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“All  trigger  catalogued” 
“All  targets  catalogued” 


15)  Control-C 

16)  Yes  and  enter 

17)  Click  on  VBS2 

18)  Escape 

19)  Abort 

20)  OK 

21)  Escape 

22)  Cancel 

23)  Cancel 

24)  Cancel 

25)  Big  red  X 

26)  Close  windows  explorer 
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APPENDIX  E.  DLI  STUDY  INFORMED  CONSENT 


Informed  Consent  Form 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  “Impact  of 
User  Interfaces  on  Virtual  Battlespace  2  Training”.  This  study  supports  a  project  to 
compare  different  user-input  interfaces  when  conducting  military  training  using 
VBS2™. 

This  study  tests  procedures  to  familiarize  users  with  the  simulation. 

Procedures.  The  study  will  consist  of  the  following: 

-  Survey  to  better  understand  the  user’s  level  of  computer  gaming 
expertise. 

-  Overview  brief  describing  the  research  project  and  VBS2™  simulations 
basics. 

-  User  training,  where  participants  will  be  able  to  move  dismount  and  drive  a 
HUMMV.  The  participants  will  also  engage  targets  with  personnel 
weapons  and  mounted  crew  served  weapons  in  the  VBS2™  simulation. 

-  User  evaluation,  where  participants  will  complete  a  short  obstacle  course 
demonstrating  their  skill  level  with  the  simulation. 

-  Post  survey  1 . 

-  10  minute  break. 

-  User  training  on  a  different  VBS2™  user-input  configuration. 

-  User  evaluation  on  a  different  VBS2^'^  user-input  configuration. 

-  Post  survey  2. 

The  study  will  take  no  longer  than  2  hours.  The  first  session  will  last  50  minutes. 
Then  there  will  be  one  10  minute  break.  The  last  session  will  take  50  minutes. 

Risks.  The  potential  risks  of  participating  in  this  study  are  not  greater  than 
minimal  risk.  The  only  foreseeable  risks  or  discomforts  associated  with  this  study 
is  the  possibility  of  participants  experiencing  simulator  sickness,  head  ache 
and/or  nausea.  This  possibility  is  no  greater  than  those  encountered  when 
playing  civilian  video  games. 

Benefits.  The  anticipated  benefit  from  this  study  is  gaining  insight  into  how  to  best 
configure  first  person  shooter  simulations  when  using  them  as  tactical  training 
devices. 

Compensation.  No  tangible  compensation  will  be  given.  A  copy  of  the  research 
results  will  be  available  at  the  conclusion  of  the  experiment.  If  you  would  like  a 
copy  of  the  results,  e-mail  Major  William  Glaser  at  wrglaser@nps.edu. 
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Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this 
study  will  be  kept  confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within 
reason,  will  be  made  to  keep  your  personal  information  in  your  research  record 
confidential  but  total  confidentiality  cannot  be  guaranteed.  No  information  which 
could  identify  a  participant  will  be  publicly  accessible.  Records  of  participation 
will  be  maintained  by  NPS  for  3  years,  after  which  they  will  be  destroyed. 
However,  it  is  possible  that  the  researcher  may  be  required  to  divulge  information 
obtained  in  the  course  of  this  research  to  the  subject’s  chain  of  command  or 
other  legal  body. 

Voluntary  Nature  of  the  Study.  Participation  in  this  study  is  strictly  voluntary,  and 
if  agreement  to  participation  is  given,  it  can  be  withdrawn  at  any  time  without 
prejudice. 

Points  of  Contact.  It  is  understood  that  should  any  questions  or  comments  arise 
regarding  this  project,  or  a  research  related  injury  is  received,  the  Principal 
Investigator,  Dr.  William  J.  Becker,  656-3963,  wibecker@nps.edu  should  be 
contacted.  Any  other  questions  or  concerns  may  be  addressed  to  the  Navy 
Postgraduate  School.  IRB  Chair,  Dr.  Angela  O'Dea,  831-656-2998, 
alodea@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been 
given  the  opportunity  to  ask  questions  and  all  the  questions  have  been  answered 
to  my  satisfaction.  I  have  been  provided  a  copy  of  this  form  for  my  records  and  I 
agree  to  participate  in  this  study.  I  understand  that  by  agreeing  to  participate  in 
this  research  and  signing  this  form,  I  do  not  waive  any  of  my  legal  rights. 


Participant’s  Printed  Name  and  Signature 


Date 


Researcher’s  Signature  Date 

WILLIAM  R.  GLASER 
MAJ,  AR  (FA57) 

NPS  MOVES  Student 
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APPENDIX  F.  DU  STUDY  APPROVED  IRB  PROTOCOL 


Naval  Postgraduate  School 

Institutional  Review  Board  (IRB) 


From; 

Via: 

To: 


Subject: 


Enel: 


1. 


2. 


3. 

4. 


5. 


6. 


President,  Naval  Postgraduate  School  MARittmo 

Chairman,  Institutional  Review  Board 
William  Becker,  MOVES  Institute 
William  Glaser 

PROJECT  STATUS:  IMPACT  OF  USER  INTERFACE  ON  VIRTUAL 
BATTLESPACE  2  {VBS2)  TRAINING 

(1)  Approved  IRB  Protocol 

The  NPS  IRB  is  pleased  to  inform  you  that  the  NPS  President  has  approved  your 
project  (NPS  IRB#  NPS.2010.0049-IR-EP7-A).  The  approved  IRB  Protocol  is 
found  in  enclosure  (1).  Completion  of  the  CITI  Research  Ethics  Training  has  been 
confirmed. 

This  approval  expires  on  01  September  10.  If  additional  time  is  required  to 
complete  the  research,  a  continuing  review  report  must  be  approved  by  the  IRB 
and  NPS  President  prior  to  the  expiration  of  approval.  At  expiration  all  research 
(subject  recruitment,  data  collection,  analysis  of  data  containing  PII)  must  cease. 

You  are  required  to  report  to  the  IRB  any  unanticipated  problems  or  serious 
adverse  events  to  the  NPS  IRB  within  24  hours  of  the  occurrence. 

Any  proposed  changes  in  IRB  approved  research  must  be  reviewed  and  approved 
by  the  NPS  IRB  and  NPS  President  prior  to  implementation  except  where 
necessary  to  eliminate  apparent  immediate  hazards  to  research  participants  and 
subjects. 

As  the  Principal  Investigator  it  is  your  responsibility  to  ensure  that  the  research 
and  the  actions  of  all  project  personnel  involved  in  conducting  this  study  will 
conform  with  the  IRB  approved  protocol  and  IRB  requirements/policies. 

After  the  experiment  is  completed  the  Principal  Investigator  will  submit  to 
the  Human  Subjects  Protection  Office,  all  signed  informed  consent 
documents,  unanticipated  problem  reports,  adverse  event  reports  and  a 
End  of  Experiment  Report.  The  Human  Subjects  Research  Office  will 
secure  these  documents  for  10  years  and  then  forward  to  the  nears  FRC. 


Angela  O’Dea  PhD 
Chair 

Institutional  Review  Board 


Daniel  T.  Oliver 
President 

Naval  Postgraduate  School 
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APPENDIX  G.  DLI  DEMOGRAPHIC  SURVEY 


1.  Please  enter  your  participant  number.  (Your  participant  number  is  the  last  4  digits  of 
your  home  or  cell  telephone  number) 

2.  Please  enter  your  age. 

3.  Are  you  a  male  or  female? 

4.  Please  indicate  your  military  service  branch  or  select  civilian. 


a. 

"Army," 

e. 

"International  Military 

b. 

"Marine  Corps," 

Member," 

c. 

"Navy," 

f. 

"Civilian" 

d. 

"Air  Force," 

g- 

"Retired  Military" 

Please 

indicate  your  military  rank. 

a. 

"El," 

h. 

"E8," 

0.  "01," 

b. 

"E2," 

I. 

"E9," 

p.  "02," 

c. 

"E3," 

j- 

"WOl," 

q.  "03," 

d. 

"E4," 

k. 

"CW2," 

r.  "04," 

e. 

"E5," 

I. 

"CW3," 

s.  "05," 

f. 

"E6," 

m. 

"CW4," 

t.  "06 

g- 

"E7," 

n. 

"CW5," 

6.  Please  indicate  your  what  category  your  MOS  falls  into. 

a.  Operations  (Combat  arms  /  war  fighter  /  combat  pilot  /  trigger  puller) 

b.  Operational  support  (Military  intelligence  /  linguist ) 

c.  Logistical  support  (Supply  /  transportation  /  maintenance  /  admin  /  legal  / 
dental) 

d.  Technical  support  (Signal  /  communications  /  computer  specialist) 

7.  Do  you  play  video  games  on  computers  (PC  or  Mac)? 

a.  "Yes/'  b.  "No" 

8.  How  many  years  have  you  played  video  games  on  a  personal  computer  (or  Mac)  on  a 
regular  basis?  (Regular  basis  is  defined  at  least  once  a  week) 

a.  _ 

9.  How  many  hours  per  week  do  you  spend  playing  video  games  on  a  personal  computer 
(or  Mac)? 

a.  _ 


-  Please  turn  over  and  complete  survey  - 
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10.  When  you  play  video  games  on  a  computer  what  do  you  play  most?  (Please  Circle) 

a.  {"Action  (Mario  Brothers,  Street  Fighter),"  "First  Person  Shooter  (Doom,  Flalo, 
Call  of  Duty) , ""Sports  (EA  Sports),"  "Action-Adventure  (Prince  of  Persia)," 
"Adventure  (Myst,  Resident  Evil  4),"  "Vehicle  Simulation  ,"  "Vehicle  Simulation 
(Racing  Games,  Flight  Sim),"  "Role-Playing  (Final  Fantasy,  Everquest,  WOW)," 
"Strategy  (War  Craft,  Command  and  Conquer),"  "Construction  and  Management 
(Simcity),"  "Life  Simulations  (The  Sims  or  Second  Life),"  "Other  (Rock  Band)" 

11.  "Do  you  play  console  video  games  at  least  once  a  week?  (Example  of  console  video 
games  are  Sega,  PS3,  Xbox,  Wii,  ect) 

a.  "Yes,"  b.  "No" 

12.  Flow  many  years  have  you  played  console  video  games  on  a  regular  basis?  (Regular  basis 
is  defined  at  least  once  a  week)  (Consoles  defined  as  Sega,  PlayStation,  Nintendo,  or 
Xbox) 


13.  Flow  many  hours  per  week  do  you  spend  playing  console  video  games?  (Console  video 
games  are  defined  as  Sega,  PlayStation,  Nintendo,  or  Xbox) 
a.  _ 


14.  When  you  play  console  video  games  what  console  do  you  prefer? 

a.  Playstation  c.  Sega 

b.  Xbox  d.  Nintendo 

15.  "When  you  play  console  video  games  what  do  you  play  most?  (Console  video  games  are 
defined  as  Sega,  PlayStation,  Nintendo,  or  Xbox)  (Type  (Example)) 

a.  "Action  (Mario  Brothers,  Street  Fighter),"  "First  Person  Shooter  (Doom,  Flalo, 

Call  of  Duty) , ""Sports  (EA  Sports),"  "Action-Adventure  (Prince  of  Persia)," 
"Adventure  (Myst,  Resident  Evil  4),"  "Vehicle  Simulation  ,"  "Vehicle  Simulation 
(Racing  Games,  Flight  Sim),"  "Role-Playing  (Final  Fantasy,  Everquest,  WOW)," 
"Strategy  (War  Craft,  Command  and  Conquer),"  "Construction  and  Management 
(Simcity),"  "Life  Simulations  (The  Sims  or  Second  Life),"  "Other  (Rock  Band) 

16.  If  you  had  a  choice,  would  you  prefer  play  a  video  game  on  a  computer  with  keyboard 
and  mouse  or  would  you  prefer  to  play  the  video  game  on  a  console  gaming  system  with 
a  game  controller,  your  answer  is  dependent  on  the  type  of  game,  assume  you  are 
playing  a  first  person  shooter  game  such  as  Doom,  Flalo,  or  Call  of  Duty.  I  would  prefer 
to  play  with: 
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a.  "Keyboard 
and 

Mouse/' 


b.  "Game 


Controller, 


c.  "I  have  no 
preferenc 
e" 


17.  "Do  you  consider  yourself  a  gamer?" 

a.  "Yes,"  b.  "No" 

18.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for 
each  "Computer  based  simulation  can  be  an  effective  training  tool  for  unit  tactical 
training." 

a.  "Strongly  Disagree," 

b.  "Moderately  Disagree," 

c.  "Mildly  Disagree," 

d.  "No  Opinion," 

e.  "Mildly  Agree," 
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APPENDIX  H.  DLI  POST  EXERCISE  SURVEY 


1.  Please  enter  your  participant  number.  (Your  participant  number  is  the  last  4  digits  of 
your  home  or  cell  telephone  number) 
a.  _ 


2.  Please  enter  your  experimental  group? 

a.  Group  4  used  PS3  and  Head  Tracker  on  the  first  run  and  PS3  only  on  second  run 

f 

b.  Group  5  used  PS3  and  Head  Tracker  on  the  first  run  and  Keyboard  and  Head 
Tracker  on  second  run. 

3.  Which  user  input  device  did  you  use  on  FIRST  run? 

a.  "Keyboard  and  mouse,"  b.  "PS3  Game  Controller" 

4.  Did  you  use  the  Naturalpoint  Head  Tracker  on  FIRST  run? 

a.  Yes,  I  used  the  Head  b.  No,  I  did  not  use  the  Head 

Tracker  Tracker 


5.  "This  question  is  in  regards  to  the  VBS2  game  software  only,  not  the  equipment 

configuration.  Rate  your  agreement  with  the  following  statement.  VBS2  could  provide 
valuable  military  training. 


a. 

"Strongly  Disagree," 

e. 

"Mildly  Agree," 

b. 

"Moderately  Disagree," 

f. 

"Moderately  Agree," 

c. 

"Mildly  Disagree," 

g- 

"Strongly  Agree" 

d. 

"No  Opinion," 

6.  How  would  you  rate  your  last  used  VBS2  user  input  configuration? 

a. 

"Very  Bad," 

e. 

"OK," 

b. 

"Bad," 

f. 

"Good," 

c. 

"Poor," 

g- 

"Very  Good" 

d. 

"No  Opinion," 

7.  Do  you  think  your  last  used  VBS2  user  input  configuration  was  good  enough  to  conduct 

valuable  military  training? 

a. 

"No," 

b. 

"Yes" 

-  Please  turn  over  and  complete  survey  - 
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8.  "Rate  your  confidence  in  doing  the  following  tasks  in  the  VBS2  with  your  last  used  VBS2 
input  configuration.  (1  means  you  are  HIGHLY  confident  and  5  means  you  are  NOT 
confident.) 

a.  Controlling  Dismounted  movements  (walking,  running,  lying  in  prone) 


1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

Observing  your  immediate  environment  while  c 

lismounted 

1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

c.  Driving  a  vehicle  (in  general). 


1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

d. 

Observing  your  immediate  environment  while  c 

Iriving 

1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

e. 

Controlling  your  M4  rifle  to  accurately  engage  targets 

1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

f. 

Finding  your  targets  when  dismounted  with  your  M4  Rifle 

1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

g- 

Controlling  your  M2  heavy  machinegun  to  accurately  engage  targets. 

1  -  Highly 

Confident 

2 

3 

4 

5  -  Not  Confident 

h.  Finding  you  targets  when  mounted  with  your  M2  heavy  machinegun 


1  -  Highly 

2 

3 

4 

5  -  Not  Confident 

Confident 

9.  Please  comment  on  how  to  improve  the  simulation  you  just  used. 
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APPENDIX  I.  DLI  POST  EXERCISE  SURVEY  ADDITION 


1.  Please  enter  your  participant  number: _ 

2.  When  performing  the  following  tasks  in  VBS2,  which  user  input  device  would  you 
prefer? 

a.  Dismounted  maneuver  -  body  movements  in  general  (walking,  running,  lying  in 


prone) 


Keyboard  and  Mouse 

No  Preference 

PS3 

b.Situational  Awareness  w 

hile  moving  dismounted. 

Keyboard  and  Mouse 

No  Preference 

PS3 

c.  Driving  a  vehicle  (in  general). 

Keyboard  and  Mouse 

No  Preference 

PS3 

d. Making  siow  tight  turns  (avoiding  obstacies). 

Keyboard  and  Mouse 

No  Preference 

PS3 

e. Making  high  speed  turns  (staying  on  the  road). 

Keyboard  and  Mouse 

No  Preference 

PS3 

f.  Combat  parking  vehicle  (Backing  up  vehicle). 

Keyboard  and  Mouse 

No  Preference 

PS3 

g. Situational  Awareness  w 

hiie  driving. 

Keyboard  and  Mouse 

No  Preference 

PS3 

h.Ability  to  shoot  your  rifle  (in  general). 

Keyboard  and  Mouse 

No  Preference 

PS3 

i.  Ability  to  acquire  (find)  targets  with  your  rifle. 

Keyboard  and  Mouse 

No  Preference 

PS3 

j.  Ability  to  accurately  engage  targets  with  your  rifle  (precision). 

Keyboard  and  Mouse 

No  Preference 

PS3 

k.  Ability  to  fire  your  mounted  heavy  machinegun  (in  general). 

Keyboard  and  Mouse 

No  Preference 

PS3 

I.  Ability  to  acquire  (find)  targets  with  your  mounted  heavy  machinegun. 
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Keyboard  and  Mouse 

No  Preference 

PS3 

m.  Ability  to  accurately  engage  targets  with  your  mounted  heavy  machinegun 
(precision). 

Keyboard  and  Mouse 

No  Preference 

PS3 
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APPENDIX  J.  DLI  STUDY  MAPS 


1.  start  Point 

2.  M4  Training  Range 

3.  M2  Training  Range 

4.  Serpentine 

5.  End  of  Training  Course 


Figure  26.  Training  course  map  for  DLI  study 


Figure  27.  Training  course  imagery  for  DLI  Study 
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Evaluation 

Course 


1.  Start  point 

2.  Serpentine 

3.  Battle  Position  1 

4.  Battle  Position  2 

5.  Dismount  Point 

6.  Shoot  /No  Shoot  Lane 
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Figure  28.  Evaluation  course  map  for  DLI  Study 


Figure  29.  Evaluation  course  imagery  for  DLI  Study 
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APPENDIX  K.  DU  CHEAT  SHEETS 


Keyboard  Hot  Key  Sheet 


Command 

PS3  Control 

Look 

Look  Up 

Mouse  Forward 

Look  Down 

Mouse  Backward 

Look  Left 

Mouse  Left 

Look  Right 

Mouse  Right 

Move 

Forward 

W 

Backward 

S 

Strafe  Left 

A 

Strafe  Right 

D 

Drive  Slow  Forward 

Q 

Additional  Moves 

Run 

Left  Shift  and  W 

Crouch 

X 

Prone 

Z 

Stand  Up 

c 

Weapons 

Fire  Weapon 

Left  Mouse  Button 

Use  Sights 

V 

Flold  Breath 

Right  Mouse  Button 

Reload 

R 

Head  Tracker 

Center  Tracker 

F12 

Pause 

F9 

Other 

Interact  With  Vehicle 

U 

Freelook 

Left  Alt 

Compass 

Y 

GPS 

G 
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PS3  Hot  Key  Sheet 


Command 

PS3  Control 

Look 

Look  Up 

Right  Stick  Up 

Look  Down 

Right  Stick  Down 

Look  Left 

Right  Stick  Left 

Look  Right 

Right  Stick  Right 

Move 

Move  Forward 

Left  Stick  Up 

Move  Backward 

Left  Stick  Down 

Move  (Strafe)  Left 

Left  Stick  Left 

Move  (Strafe)  Right 

Left  Stick  Right 

Additional  Moves 

Run 

R2 

Crouch  /  Up 

Push  Right  Stick  Button 

Prone  /  Stand  Up 

Push  Left  Stick  Button 

Weapons 

Fire  Weapon 

R1 

Use  Sights 

LI 

Flold  Breath 

L2 

Reload 

Triangle 

Other 

Interact  With  Vehicle 

Select 

Freelook 

Square 

Compass 

Right 

GPS 

Left 

Head  Tracker 

Center  Tracker 

F12 

Pause 

F9 

126 


PS3  Hot  Key  Sheet 


Left  Stick 


Right  Stick 
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APPENDIX  L.  AAR  EVALUATION  INSTRUCTIONS 


1.  Turn  On  Computer 

2.  Login 

•  Ctrl-Alt-Del  x  2 

•  User  Name:  administrator 

•  Password:  DVTEM90build2 

•  OK 

3.  Move  all  aar  files  to  be  evaluated  from  AARs  for  the  “AARs  for  Wesley 
Review”  Folder  to  c:\documents  and  settings\Administrator\My  documents\ 
VBS2™  \aar.  Note:  do  not  move  all  in  at  one  time  I  recommend  you  move 
them  in  blocks  of  10  or  12  as  not  to  get  aars  and  participant  #  confused. 

4.  Open  “  VBS2™  Administrator”  found  on  desktop  or  in  a  folder  on  the 
desktop. 

5.  click  on  After  Action  Review  on  left  side  of  initial  menu  screen 

6.  Click  on  Sahrani 

7.  Click  on  the  aar  you  wish  to  review.  It  should  show  the  same  files  that  you 
moved  in  the  aar  folder  in  step  3. 

8.  Click  on  view  to  hide  notepad  if  necessary. 

9.  This  provides  you  a  view  of  the  map  centered  on  Corazo.  Cur  mission  takes 
place  far  south  of  Corazo.  Use  the  scroll  wheel  to  scroll  out.  Place  the  cursor 
over  Iguana  (in  the  south)  and  scroll  in.  Scroll  in  on  the  one  blue  icon  north 
east  of  the  town  between  two  green  vehicle  icons.  Right  click  on  the  blue  icon 
so  it  flashes.  It  should  say  Rifleman  -  M16A4  -  (Marine). 

10.  Click  on  View  and  then  3D  Camera  View 

1 1 .  Right  click  on  the  green  Hummwv  and  choose  lock  camera.  Right  click  and 
drag  to  change  your  perspective.  Scroll  in  and  out  to  change  your  distance 
from  the  Hummwv. 

12.  Click  the  play  arrow  to  start  the  aar.  Wait,  this  takes  a  few  seconds.  Your 
camera  view  should  move  with  the  moving  vehicle  if  you  properly  locked  the 
camera  view  on  the  vehicle. 

13.  Record  the  Time  when 

•  The  vehicle  starts  to  move,  this  is  usually  not  zero. 

•  Enters  the  serpentine,  (hits  the  shadow) 

•  Count  the  number  to  times  the  Hummwv  hits  a  serpentine  wall 

•  Exit  the  serpentine 

•  Enters  Battle  Position  1,  (hits  the  shadow). 

•  Hits  the  first  truck  (burst  into  flames),  hits  the  second  truck,  &  hits 
the  third  truck 

•  Count  the  number  to  times  the  Hummwv  hits  an  object,  structure, 
scenery,  street  sign,  street  light,  other  vehicle,  or  all  four  wheels 
leave  the  black  top  road. 

•  Enters  Battle  Position  2 
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•  Hits  the  first  truck  (burst  into  flames),  hits  the  second  truck,  &  hits 
the  third  truck 

•  Count  the  number  to  times  the  Hummwv  hits  a  object,  structure, 
scenery,  street  sign,  street  light,  other  vehicle,  or  all  four  wheels 
leave  the  black  top  road. 

•  End  of  mounted  course  /  beginning  of  combat  parking  lane. 

•  Score  combat  parking  lane 

•  End  of  combat  parking  lane  /  beginning  of  the  shoot  /  no  shoot 
scenario  (crossing  the  green  arrows) 

14.  When  the  participant  dismounts  pause  the  AAR  by  clicking  on  the  play  arrow. 
Move  cursor  over  the  soldier  and  right  click.  Choose  lock  camera.  Adjust 
camera  as  necessary. 

15.  The  shoot  house  bad  guys  targets  (green  with  yellow  “X”)are  Left  alley.  Right 
alley.  Left  Alley,  Left  Alley.  Record  how  many  bad  guy  targets  are  hit  (should 
be  4).  Record  how  many  good  guy  targets  (Blue)  are  hit  (should  be  none).  A 
blue  tracer  means  target  hit.  Record  how  many  total  rounds  expended. 

•  Enter  Time 

•  Bad  guy  targets  hit  (green) 

•  Good  guy  targets  hit  (blue) 

•  Total  rounds  expended 

•  Exit  Time 

16.  Dismount  portion 

•  Enter  time  (start  ramps  -  cross  green  arrow) 

•  Score  ramps.  Use  the  below  as  a  scoring  matrix.  If  the  player  falls 

off  the  ramp  add  the  points  below.  The  lower  the  score  the  better. 
The  worst  score  is  a  10.  For  example,  fall  off  Ramp2a  and 
RampSb  =  6  pts  _ _ _ _ 


Rampi 

Ramp2a 

Ramp2b 

Ramp3a 

Ramp3b 

Ramp3c 

3 

3 

2 

4 

3 

2 

•  End  Time  Dismount  /  Start  time  M4  range  (crosses  green  arrows) 

17.  Range 

•  Enter  Start  time  M4  Range  (crosses  green  arrows)  -  same  as 
above 

•  Insurgents  hits  (should  be  4) 

•  Red  popup  targets  (should  be  3) 

•  Total  rounds  expended 

•  End  time  of  last  target 

18.  Overall  Grade 

I  need  you  to  assign  a  subjective  overall  grade  to  the  run. 

•  A  =  user  looks  as  if  he  could  focus  100  percent  of  his  effort  on  the 
military  training  task  and  0  percent  on  using  the  interface. 
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Definitely  could  accomplish  the  training  objectives  using  assigned 
user  interface  in  VBS2™. 

•  B  =  user  looks  as  if  he  could  focus  75  percent  of  his  effort  on  the 
military  training  task  and  25  percent  on  using  the  interface.  Should 
be  able  to  accomplish  the  training  objectives  using  assigned  user 
interface  in  VBS2™. 

•  C  =  user  looks  as  if  he  could  focus  50  percent  of  his  effort  on  the 
military  training  task  and  50  percent  on  using  the  interface 

•  C  =  25  percent  /  75  percent  -  ect. 

•  F  =  0  percent  / 100  percent  -  Definitely  could  not  accomplish  the 
training  objectives  using  assigned  user  interface  in  VBS2™. 

19.  Exit 

•  Click  file  and  chose  exit 

•  Load  next  AAR  or  choose  Cancel  to  go  to  main  menu 
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